The Incrementality checklist – what to know before getting started
By: Thi Thumasathit, Head of Product – Campaigns in Product and Design (Sydney)
Over the course of this Incrementality series, we have equipped you with knowledge about the Incrementality methodology, introduced the ‘Would Have Seen (WHS)’ and how and what Incrementality can measure. Now the time has come to dish out some final tips on how to really get yourselves situated with Incrementality.
Ask your digital marketing platform if they have an Incrementality solution.
Because the Incrementality methodology is so deeply embedded within a platform’s campaign/ad selection process, Incrementality solutions are platform-specific. It’s not as simple as going out and renting or buying an Incrementality solution and using it for measuring and testing campaign effectiveness across all your digital marketing platforms.
Be patient.
Like all scientific experiments, Incrementality studies require time and data to produce concrete results. Work with your digital marketing platform to estimate how long it will take for your Incrementality study to reach statistical significance.
Define your key questions.
Define what key questions you’re trying to answer with an Incrementality study.
Most brands start off with the highest level question: what incremental impact can this digital marketing platform have on overall revenue?
Some brands ask more specific questions: what incremental impact can this digital marketing platform have on one specific customer segment or a particular product?
It’s important to identify what these specific questions are and answer them upfront:
– The questions might affect how campaigns are structured. For example, if your really want to know the impact a given platform is having during the holiday season, you need to make sure you have campaigns running on that platform during the holiday season.
– The questions might be limited by what data is/isn’t tracked in the platform. For example, if the platform doesn’t track gender, you can’t ask gender-related questions.
– Keep in mind that the more questions you ask, the smaller the data sets become, and the longer it will take to reach statistically significant answers. Focus on the most important questions at hand, and ask yourself how you would use this information if you knew the outcome already.
Recognize what Incrementality is, and is not.
Incrementality is a platform-wide solution that helps to measure the impact/effectiveness of an entire platform but with time we will be able to measure at an ever greater granularity.
Incrementality is not a solution to measure the relative effectiveness of specific creatives. A/B split testing is a more appropriate tool for testing specific creatives.
Similarly, Incrementality is not a solution to measure the relative effectiveness of campaigns with similar objectives where the users they are both targeting (strongly) overlap. For example, if two campaigns share the same objective — e.g., to acquire new customers — and users are exposed to both campaigns, it would be difficult to truly isolate the performance of each campaign individually in the results.
“Like all scientific experiments, Incrementality studies require time and data to produce concrete results.”
Verify the definition of the Would Have Seen.
The WHS are truly identical to users who were exposed to a brand’s ad. Make sure the point of determining if a user is/isn’t a WHS occurs after:
– The platform determines if the user is/isn’t in an audience inclusion/exclusion list
– The platform determines if the user is/isn’t part of the brand’s target audience
– The brand bids on the user
– The brand wins the auction to show an ad to the user
Only then can a user be deemed a WHS. If any of the above conditions aren’t met, the Incrementality solution may introduce bias into the WHS and materially impact the accuracy of the Control group.
Ask for the details.
Any good incrementality solution should not be a black box, where you take the results at blind faith. Ask for full documentation of the platform’s Incrementality testing methodology, and get the details behind which conversions were test vs. control users along with a deeper understanding of the mechanism that is used to split between test vs control if that interests you. Note, however, that privacy regulations may limit the amount or type of data that digital marketing platforms can share with you.