Incrementality Testing

Incrementality testing measures the true impact of your marketing campaigns by comparing what happens with your campaigns (test group) versus what would have happened without them (control group). It's a scientific method that isolates the causal effect of your marketing by creating two comparable groups—one exposed to your campaigns and one not—then comparing their performance. This helps you identify which campaigns drive genuine incremental value beyond organic activity, rather than simply replacing organic installs with paid ones. The goal is to answer the question: 'Would this user have installed my app or made a purchase even if they hadn't seen my ad?'

Attribution shows you which campaigns users came from, but it doesn't tell you if those users would have installed your app organically anyway. It's a correlation, not causation. Incrementality testing reveals the true incremental value—the additional users, revenue, or engagement your campaigns generate beyond what would have occurred naturally. Think of it this way: attribution tells you 'this user came from Campaign A,' while incrementality tells you 'this user would not have installed without Campaign A.' This distinction is crucial because paying for users you'd acquire for free wastes budget that could drive genuine growth. Many marketers discover that 20-50% of their attributed installs would have occurred organically.

Adshift data provides the attribution, user behavior, and performance metrics needed for incrementality testing. You can use install attribution to segment users into test and control groups based on campaign exposure. Leverage post-install event data to measure incremental engagement—comparing event rates between exposed and unexposed users. Analyze revenue and LTV metrics to quantify true campaign lift. Adshift's granular data allows you to segment by media source, campaign, creative, placement, and other dimensions, giving you flexibility to test specific hypotheses. The key is comparing performance between exposed and unexposed user groups using Adshift's comprehensive data across installs, events, revenue, retention, and LTV.

Organic cannibalization occurs when paid campaigns simply replace organic installs rather than generating new users. For example, if a user was browsing your app in the store and would have installed it organically, but then clicked on a paid ad, you're paying for a user you would have gotten for free. This is one of the biggest hidden costs in mobile marketing. Studies show that cannibalization rates can range from 20-50% depending on the channel and campaign. Incrementality testing helps you detect and measure this cannibalization by comparing test groups (exposed to campaigns) with control groups (not exposed). Once you identify cannibalization, you can optimize your campaigns to focus on truly incremental users—for example, by adjusting targeting to reach users who are less likely to install organically.

Yes. Incrementality testing can be conducted using aggregated data that complies with privacy regulations like iOS 14+ SKAdNetwork and Google Privacy Sandbox. The shift toward privacy-first measurement doesn't eliminate incrementality testing—it just means you work with aggregated data rather than user-level data. Adshift provides aggregated metrics on installs, events, and revenue at the campaign level that can be used for incrementality analysis. For example, SKAdNetwork postbacks provide aggregated campaign performance data that enables you to compare test and control groups at the campaign level, maintaining user privacy while still enabling incrementality testing.

Adshift data enables several incrementality testing methodologies. Geographic holdout tests compare regions with campaigns running to regions without campaigns—Adshift's geo-level attribution data makes this possible. Time-based holdout tests compare periods with campaigns to periods without, using Adshift's time-series data. User-level holdouts require setting up tests at the campaign level where campaigns are actually withheld from some users (where privacy regulations allow), then Adshift's attribution data can identify which users were exposed versus unexposed for comparison. Synthetic control is an analytical technique that uses Adshift's historical data to create control groups through statistical modeling. The key is using Adshift's comprehensive data to create statistically significant test and control groups that isolate the true impact of your marketing. Adshift's granular segmentation capabilities allow you to design tests that answer specific questions about which campaigns, channels, or strategies drive incremental value.

Reliable incrementality testing requires sufficient volume and duration to achieve statistical significance. Generally, you need enough installs in both test and control groups to detect meaningful differences—typically at least 1,000 installs per group, though this varies based on your expected lift. Test duration should be long enough to capture the full impact of your campaigns, including delayed conversions and long-term effects like retention and LTV. Most incrementality tests run for 2-4 weeks, but longer tests (6-8 weeks) provide more reliable results, especially for measuring incremental retention and LTV. Adshift's data allows you to monitor test performance in real time and determine when you have enough data for statistically significant results.

Incrementality test results should directly inform your budget allocation and campaign optimization. When you identify campaigns with high incrementality, scale them with confidence—these are driving genuine growth. When you find campaigns with low or negative incrementality (cannibalization), pause them or adjust targeting to focus on truly incremental users. Use incrementality data to optimize your media mix, shifting budget from low-incrementality channels to high-incrementality ones. Adjust creative and targeting strategies based on what drives incremental value. Adshift's real-time data allows you to monitor incrementality metrics continuously and make adjustments as you learn which campaigns drive the best incremental results. The goal is to create a feedback loop where incrementality insights inform optimization, which improves incrementality, which informs further optimization.