Get Faster Answers. Make More Confident Budget Decisions
TL;DR Summary
Summary… (click to expand)
New self-service incrementality testing in Kochava makes it possible for app marketers to create, schedule, and run pulse tests directly in their dashboard, without complicated data science teams. Configure your test by selecting the app, region, network partner, campaign, holdout schedule, and the specific events or KPIs you want to measure. Once tests complete, AI-generated insights translate statistical results into plain-language budget and optimization recommendations.
For teams navigating disagreements between last-touch attribution and marketing mix modeling, pulse testing serves as the causal tiebreaker. A real-world example: a lifestyle app used Kochava incrementality testing to resolve a significant discrepancy between their last-touch attribution (LTA) and marketing mix modeling (MMM) data, arriving at confident, evidence-backed budget decisions. Read the case study. Incrementality Testing is available now to all Kochava customers using AIM (Always-On Incremental Measurement), Kochava’s MMM solution. Contact Kochava
Self-service incrementality testing is now available inside the Kochava platform. This enables app marketing teams to create, schedule, and run incrementality tests directly, without needing to involve another third-party tool, an external agency, or a dedicated data science team. The result is faster access to statistically rigorous answers for the questions that sit at the center of every marketing budget conversation.
The Questions Every Marketing Team Should Be Asking
Let’s begin by unpacking the specific types of questions incrementality testing is built to answer.
- Is this channel actually worth the investment, or is it capturing demand that would have existed regardless of whether my ads ran?
- Is my branded search campaign driving net new users, or are those users already looking for me and finding me organically anyway?
- Are my channels working together to grow total volume, or am I cannibalizing performance across them and moving the same pool of users around?
- If I pause one channel or partner, does overall performance actually drop, or does it hold steady because the demand simply shifts elsewhere?
- When my last-touch attribution and my MMM disagree on a channel partner’s contribution, how should I best navigate the difference to guide investment decisions?
These are not hypothetical questions. They come up in budget reviews, in post-campaign debriefs, in conversations with finance teams, and in quarterly planning. Incrementality testing gives you a structured, evidence-based approach to answer them, rather than relying on instinct or assumptions. App marketing teams most effectively find their ground truth when combining incrementality testing with MMM and last-touch attribution measurement disciplines..
The Problem with Incrementality Testing (Until Now)
For years, incrementality testing has been one of the most powerful tools in a performance marketer’s toolkit, and one of the most frustrating to actually use.
In theory, the premise is simple: pause your advertising in a controlled group, keep it running for everyone else, and compare the difference. The gap in outcomes between those two groups is your true incremental lift. In practice, it has rarely been that simple.
Running a proper incrementality test often requires data science resources, precise experimental design, careful holdout group construction, and enough statistical discipline to ensure the results actually mean something. Even for teams that have those resources in-house, the process is quite time-consuming. For teams that do not, the options have typically been to bring in an outside agency or a third-party measurement vendor, add weeks (sometimes months) to the timeline, layer in additional cost, and ultimately receive results that live in a separate system, disconnected from the attribution and MMM data already inside your platform.
The result has been that incrementality testing, despite its value, often gets reserved for large campaigns, high-stakes budget decisions, or once-a-quarter exercises. It has not been the always-on, agile practice it could be.
The Incrementality Testing solution from Kochava is designed to change that.
How Kochava Incrementality Tests Works
Kochava’s incremental testing solution uses controlled pulse experiments that measure true incremental lift, backed by statistical confidence and AI-generated budget optimization insights. Test creation, scheduling, execution, and AI-generated recommendations are all within the existing Kochava dashboard you use for marketing mix modeling insights.
What You Can Configure for an Incrementality Test
When creating a new pulse test in Kochava, here are the settings you can configure:
- Test Name: Give your experiment a clear, identifiable name for easy reference later.
- App Selection: Choose the specific app you want to measure incrementality for.
- Region: Scope your test to a country or a specific geographic region globally. Pulse testing works globally for any market, app, or budget, unlike geo-testing, which requires large addressable markets and is typically limited to the US.
- Network Partner: Select the specific ad network or partner you want to test.
- Campaign: Target the exact campaign you want to measure.
- Start Date and Holdout Schedule: Set when the first holdout period begins and define your preferred cadence.
- Holdout Period Duration: Choose the length of each on/off period: five days, six days, or seven days.
- Events to Measure: Select which in-app events or KPIs should be assessed as part of the incrementality evaluation. Whether you care about installs, registrations, purchases, or a custom event, you define what success looks like for your test.
Before the test is created, you will see a full Test Summary that lets you review and confirm every configuration parameter. Once you are satisfied everything is set correctly, you create the test with a single click.
Use Cases: What Can You Actually Test?
Incrementality testing is most powerful when applied to specific, well-scoped questions. Here are some use cases app marketers are using pulse tests for:
Validating a High-Spend Network Partner
You have a channel that consistently shows strong numbers in your attribution reporting. Before you double down on that budget, run a pulse test to confirm those conversions are truly incremental and not users who would have converted organically regardless of the ad exposure.
Evaluating a New Channel Before Scaling
When you are testing a new network or ad format, incrementality testing gives you a clean read on whether early results represent real lift or simply coincidence. That evidence makes scaling decisions far more defensible.
Resolving Attribution Model Disagreements
Last-touch attribution and MMM sometimes tell very different stories about which channels are performing. Incrementality testing serves as the tiebreaker: a causal measurement layer that cuts through the noise (as illustrated in this case study).
Identifying Ad Fatigue
If a test reveals negative incrementality, meaning your ads drove fewer conversions than would have happened organically, that is also valuable intelligence. It can signal poor creative, the wrong audience, or ad fatigue. AI insights will recommend pausing and reallocating budget.
Informing Seasonal Budget Strategy
Run pulse tests ahead of peak periods to understand which channels have the highest true incremental ROAS before you commit to larger buys.
Reviewing Your Results
Once a test has run its course, the incrementality dashboard gives you a clear view of past and active tests. From the results view, you can see the status of each test (active, completed, or scheduled), along with start dates and reporting dates. Clicking into a completed test surfaces the full results view, which is organized into several distinct layers.
Key Performance Stats
With the insight summary, you will find the headline numbers for the test:
- Incremental lift percentage: how much lift the campaign generated above the organic baseline
- Increased conversions above baseline: the raw number of additional conversions attributable to the campaign, beyond what would have occurred organically
- Cost per incremental conversion: spend efficiency measured against only the conversions the campaign actually caused, across all measured events
- Statistical power and p-value: two metrics that together tell you how reliable the result is. Statistical power (ideally above 80%) reflects whether the test had enough data to detect a real effect. The p-value (ideally below 0.05) reflects the probability that the observed lift happened by chance rather than because of the campaign. Together, they tell you whether you can act on the test results with confidence.
AI Insight Summary
Kochava generates an AI-powered plain-language summary of the test. Rather than leaving teams to interpret raw statistical output on their own, this summary surfaces the key takeaways directly: which direction the data points, how confident the result is, and what it means for spend decisions and optimization next steps. For marketers running multiple tests across different channels, partners, or regions, this makes it significantly faster to move from data to decision.
Test Period Schedule
A clear timeline of the test structure is displayed, showing the on and off periods across the test window, so you can see exactly when holdout blocks were active and how the experimental design maps to the results.
Attribution Time Series
One of the most useful views in the results dashboard is the attribution time series chart. It plots three lines across the test window: last-touch attribution, MMM attribution, and the incremental outcome trend. The holdout periods are visually marked within the chart, making it easy to see how each measurement approach responded during active and paused ad windows. This view is particularly valuable when last-touch and MMM are telling different stories about a channel, because the incremental line gives you a causal read alongside both.
Weekly Results Breakdown
For each event KPI included in the test, a weekly breakdown table shows performance across the test periods in detail. This gives teams the granularity to spot patterns across specific weeks, understand how lift developed or changed over time, and validate results at the event level rather than relying solely on aggregate numbers.
Case Study: How One Lifestyle App Used Incrementality to Break a Tie
Sometimes your last-touch attribution data and your marketing mix model disagree, and when they do, the right next step is not to guess which one to trust.
A lifestyle app marketing team recently found themselves in exactly that situation. Their last-touch attribution reporting and their MMM analytics were telling meaningfully different stories about channel performance. That gap made confident budget decisions difficult. Rather than arbitrarily favoring one signal over the other, they turned to incrementality testing to get a causal read on which channels were actually driving growth.
The results gave their team the evidence they needed to act decisively, reallocating budget with confidence backed by experimental data rather than model assumptions.
You can read the full story here: Lifestyle App Case Study: Using Incrementality to Resolve Attribution and MMM Discrepancies
Turning Results into Better Decisions
Running a test is only the beginning. The real value of incrementality testing is in how you use the results to change what you do next. A few examples of how marketers are putting pulse test findings to work:
Scenario 1: The Surprise Underperformer
A marketer runs a pulse test on a mid-tier network that has always shown solid numbers according to last-touch attribution. The test reveals minimal incremental lift. Decision: reallocate that budget to channels with proven causal impact, rather than channels that simply show up in attribution paths because they reach users already likely to convert.
Scenario 2: The Confirmed Winner
A brand new network partner shows promising early attribution results. The pulse test confirms high incremental lift at statistically significant confidence. Decision: scale the campaign with conviction, backed by causal evidence rather than correlation alone.
Scenario 3: The Regional Insight
A global app runs the same campaign across two regions. Attribution looks similar. Incrementality results tell a different story: one region shows strong lift, the other shows near-zero. Decision: increase investment in the high-lift region, reduce or pause in the low-lift region, and investigate whether creative or audience quality is driving the gap.
In each scenario, the incremental test result provides evidence that is grounded in actual behavior, not attribution model assumptions alone. That makes it easier to act with confidence, regardless of what the data shows.
Get Started
Incremental testing is available to all Kochava customers licensing AIM (Always-On Incremental Measurement). Log in to your Kochava dashboard, navigate to the Incrementality Testing section, and create your first test in minutes.
If you have questions about methodology, test design, or how to interpret your results, contact your Client Success Manager or email support@kochava.com.
Interested in exploring Kochava for MMM and incrementality testing ? Request a huddle with our team.
Frequently Asked Questions
What is incrementality testing and why does it matter for app marketers?
Incrementality testing is a method of measuring whether your advertising is causing conversions, or simply reaching users who would have converted anyway. Traditional attribution models, including last-touch and even multi-touch methods, measure correlation: an ad was shown, an install happened. Incrementality testing introduces a controlled holdout period or group to measure what happens in the absence of advertising, then compares that against the exposed group. The difference is your true incremental lift. For app marketers managing budgets across multiple networks and channels, knowing which spend is genuinely driving growth versus which is claiming credit for organic behavior is the difference between efficient scaling and wasted spend.
How does Kochava incrementality testing work, and how long does a test take to run?
Kochava incrementality testing uses an on/off pulse methodology: advertising runs normally for a defined period (the “on” window), then is paused for a holdout period (the “off” window), and results are compared across both periods to measure the incremental impact of the campaign. Marketers configure the test directly in the Kochava dashboard by selecting the app, region, network partner, campaign, start date, holdout duration (five, six, or seven days per period), and the specific in-app events or KPIs they want to measure. A test summary screen confirms all parameters before launch. Results, including AI-generated plain-language recommendations and full statistical detail, are available directly in the incrementality dashboard once the test completes and the final attribution window period is complete.
What is the difference between incrementality testing and marketing mix modeling in Kochava?
Marketing mix modeling (MMM) and incrementality testing answer related but distinct questions. MMM takes a top-down view: using historical spend and outcome data across channels to model the relative contribution of each channel to overall results. It is excellent for strategic budget allocation and understanding macro-level channel efficiency over time. Incrementality testing takes a bottom-up, experimental view: isolating a specific campaign and network, running a controlled experiment to measure causal lift for that specific variable. The two are complementary. When MMM and last-touch attribution disagree on a channel’s contribution (which happens often), an incrementality pulse test provides the causal evidence needed to resolve the conflict and make a confident budget decision. Kochava is the only platform that can give marketers all three signals, LTA, MMM, and incrementality, in a single integrated dashboard.
How should marketers use incrementality test results to make better budget decisions?
Incrementality test results surface in three typical patterns, each with a clear decision path. First, high confirmed lift: the test shows that the channel or campaign is driving genuinely incremental conversions at statistical significance. The right move is to scale with confidence. Second, low or negligible lift: the campaign shows minimal incremental impact, meaning conversions attributed to that channel are largely organic or driven by other influences. The right move is to reallocate that budget toward higher-lift channels. Third, negative lift: advertising may actually be suppressing conversions that would have occurred naturally, often a signal of creative fatigue, audience overlap, or poor targeting. The right move is to pause the campaign and investigate. Kochava’s AI-generated insights translate these outcomes into specific, plain-language recommendations so marketers do not need to interpret raw statistics themselves. The platform handles the analysis; the marketer acts on the recommendation.
