The "growth at all costs" mindset has undergone a fundamental shift. For modern brands and sellers, the future is in a sustainable approach focused on profitable growth.
As costs rise and competition intensifies, brands need better ways to measure the effectiveness of their marketing across the shopper journey. The solution isn’t straightforward: Nearly half of marketers say measuring true return on investment is their biggest challenge—but it can be done.
How? With incrementality testing. This powerful approach reveals the causal impact of your marketing efforts by comparing results between test and control groups. While traditional attribution models struggle with today's complex customer journeys, incrementality testing offers clarity on which marketing dollars actually contribute to your bottom line.
The New eCommerce Reality
Today's sellers are facing challenges that make profitable growth harder than ever before:
- Rising costs: Marketing budgets are on the rise again—however, they must stretch across more channels while advertising costs on platforms like Amazon continue to climb. Plus, Amazon alone welcomed more than 900,000 new sellers in 2024, creating heavy pressure on established brands.
- Maintaining profitability: Many brands have increased prices by 5-7% to protect margins, but pricing adjustments alone aren't enough. The most successful sellers now implement full-funnel strategies and testing different channels rather than pouring their entire budget into a single channel (e.g. Amazon ads).
- Misleading metrics: Standard attribution models often fail to provide a complete picture of your marketing effectiveness. Roadblocks like double attribution and platforms’ preferential selection for familiar traffic make it hard to suss out causality and measure your marketing plan’s long-term impact.
TL;DR Finding and protecting profitability is a huge challenge for brands of all sizes. Without a clear understanding of which marketing dollars truly drive profits, even seasoned sellers risk wasting their budget on ineffective campaigns while neglecting marketing channels that could deliver substantial growth.
Understanding Incrementality: The Foundation
Before we dive into the framework, let’s quickly level-set on what incrementality testing is, and why eCommerce brands should care.
What Is Incrementality Testing and Why Does it Matter?
Incrementality testing measures the true impact of a marketing campaign by comparing results between two groups:
- An experimental group exposed to your campaign
- A control group not exposed to the campaign
It’s a controlled experiment that reveals which campaigns actually influence conversions (causation) rather than simply happening at the same time (correlation). In other words, it answers the critical question, "Would this customer have purchased anyway, even without seeing my ad?"
Understanding the true ROI of each campaign helps you distribute your ad spend more effectively, investing in the specific campaigns and tactics that deliver genuine growth and profitability.
Building Your Incrementality Testing Framework: A Step-by-Step Action Plan
Incrementality testing might sound complicated, but simplicity is key. Here’s a step-by-step framework to build a sustainable approach that gives you clear, reliable insights without trying to do too much at once.
Step 1: Choose a Test Objective
As always, identifying a clear objective is a crucial first step. Whether you’re testing Amazon PPC, email, or social media campaigns, start by defining exactly what you’re measuring—for example: conversions, ROAS, or new customer acquisition.
Here are some tips to pinpoint the right objectives:
- Focus on a single question. Rather than trying to test everything at once, identify the most pressing question about your marketing performance.
- Choose KPIs that impact profitability. Look beyond surface metrics like impressions or clicks, and instead focus on revenue, margin, or customer lifetime value.
- Connect objectives to business goals. If your goal is profitability, make sure your test measures true incremental profit, not just sales volume.
- Think outside the sales box. Incrementality can measure other valuable outcomes like offline channel impact and messaging effectiveness.
Step 2: Select a Campaign or Channel to Test
Start with high-impact channels like Amazon Sponsored Display or Walmart Sponsored Products, where small improvements can yield big returns. Aim to strike a balance between statistical validity and practical implementation.
To get the most out of every campaign:
- Prioritize high-spend channels. The media channels where you spend the most have the greatest potential for optimization and savings.
- Consider platform capabilities. Some ad platforms like Facebook and Google offer built-in testing tools that simplify implementation.
Step 3: Select Your Testing Methodology
There are many incrementality testing methods that work for eCommerce. The right one depends on your platform, budget, and the questions you're trying to answer.
Common testing methodologies include:
| Method | Description | Use cases |
| A/B testing | Create two campaigns with only one different variable to measure its impact | Comparing creative variations or ad formatsTesting pricing or promotions |
| Conversion lift | Compare conversion rates between exposed and unexposed users | Measuring brand campaign effectivenessTesting targeting strategiesEvaluating overall channel performance |
| On/off testing | Temporarily pause ads to measure campaign performance differences | Evaluating the overall impact of a channelEstablishing baseline organic performance |
| Geographic testing | Run campaigns in certain regions while using others as holdout regions | Measuring omnichannel impact (online + offline)Testing broad awareness campaigns |
| Time-based testing | Compare performance during campaign periods vs. non-campaign periods | Measuring the impact of limited-time promotionsEvaluating long-term brand impact |
| Ghost ads | Show placeholder creative to control users instead of actual ads | Testing competitive conquesting campaignsEvaluating precise platform-specific elements |
| Intent-to-Treat (ITT) | Analyze based assigned group, even though some control users may still see ads organically | Overcoming selection bias in incrementality resultsMeasuring true, real-world impact on complex audiences |
| Synthetic control | Use algorithms to create artificial control groups with historical data | Testing when creating true control groups is impracticalEvaluating long-running campaigns retroactively |
Here are some strategies to help you choose the right testing methodology:
- Calculate your required sample size. Use free tools like this A/B test calculator to determine how many users you need for statistically significant results.
- Consider technical feasibility. Select methodologies your team can properly implement with available resources and expertise.
- Validate results with a second method. When possible, test the same question with a different approach to validate your findings.
Step 4: Create Test and Control Groups
For valid incrementality testing, you need properly structured comparison groups. If you’re evaluating impact across channels, make sure your approach accounts for the unique characteristics of each platform.
The following strategies can help:
- Measure cross-channel incrementality. Use multi-touch attribution to assign fractional credit to each touchpoint in the customer journey.
- Consider seasonality. If testing during holiday periods, ensure both groups experience similar seasonal demand patterns.
- Verify group similarity before testing. Make sure the groups are statistically similar in terms of demographics and behavior (conversion rate, AOV).
- Document control group parameters. Clearly define what marketing activity your control group will and won't be exposed to during the test.
Step 5: Launch and Monitor Your Test
During your test, closely track your results and avoid making other marketing or product changes that could weaken your conclusions.
Here are a few specific areas to keep an eye on:
- Set up proper tracking. Most eCommerce sellers should aim to capture real-time data from both platform-specific metrics and cross-platform behavior.
- Monitor for test contamination. Regularly check that control users aren't being exposed to test ads through other channels.
- Document external events. Record any significant marketplace changes, competitor actions, or platform updates during the test period.
Step 6: Measure Incremental Impact
After running your test, carefully compare the outcomes between the two groups to understand which elements drove new sales.
Here’s how to do it:
- Calculate true lift percentage. Subtract the control group conversion rate from the test group rate to find incremental impact.
- Translate lift into dollars. Multiply that lift percentage by total revenue to quantify the financial impact.
- Calculate statistical significance. To ensure your results weren't random chance, find your p-value and confidence intervals.
- Segment your analysis. Segment results by customer type, product category, or device to uncover deeper, more granular insights.
Step 7: Iterate and Optimize for Profitability
Finally, it’s time to translate your test results into actions that drive measurable profit improvement. For eCommerce sellers, this means moving beyond surface-level metrics to focus on what truly matters: money in your bank account.
Here are some tips to help you drive more incrementality long-term:
- Prioritize metrics in order of importance: Actual profit > customer lifetime value > new customer revenue > conversion rate > click-through rate.
- Apply insights to bidding strategies. Set bid modifiers based on incremental value rather than platform-reported conversions to avoid overpaying for customers who would have purchased anyway.
- Implement media mix modeling (MMM). This statistical analysis forecasts how adjusting marketing spend across channels will impact overall ROI.
- Target the right products for each campaign objective. Some products drive new customer acquisition while others support cross-selling.
- Continuously refine target audiences. Use incrementality measurement insights to develop custom audience segments that deliver the highest incremental value, not just the highest conversion rates.
Incrementality testing isn't a one-time exercise, it’s an ongoing process that helps you continuously improve and refine. Setting up a system to measure incrementality—i.e., the true causal relationships between your marketing strategy and business outcomes—will give you a serious advantage over competitors still relying on basic, correlation-based metrics.
Common Challenges and Solutions
Implementing incrementality testing does come with challenges. Here are a few common hurdles and some solutions to help make sure you end up with accurate, actionable test results.
Data Fragmentation
🚩 Challenge: Marketing data is scattered across platforms and reported differently, making it hard to get a unified view of results.
🎯 Solution: Use API connections to automatically pull data from different platforms into a single dashboard. Trellis consolidates multiple data sources including Amazon Marketing Cloud and Walmart Connect.
Dynamic Marketplace Environments
🚩 Challenge: Changing algorithms and market conditions can affect test results, making it difficult to isolate the true impact of your campaign.
🎯 Solutions: Document external factors that might influence results, including competitor price changes, inventory fluctuations, and marketplace platform updates. Conduct continuous testing rather than one-off experiments to create a rolling baseline and ongoing validation.
Seasonality and Consumer Trends
🚩 Challenge: Seasonal patterns can significantly skew test results if not properly accounted for in your measurement approach.
🎯 Solutions: Compare year-over-year results during similar seasonal periods rather than back-to-back timeframes. You can also use season-adjusted baselines when calculating incremental lift.
Competition and Pricing Pressure
🚩 Challenge: Allocating budget to test incrementality is tough when you’re already balancing competitive pricing with rising ad costs.
🎯 Solution: Start testing in baby steps. Decide how much you can allocate for incrementality testing and stick to it—and remember, executing your testing plan with care will produce valuable insights that improve profitability and save you money in the long run.
Illuminate Your Marketing Impact With Incrementality Testing
For today’s eCommerce brands, understanding the true impact of advertising spend isn't a pipe dream; it's an essential piece of the profitability puzzle.
Incrementality testing gives you the clarity to make informed decisions about where to invest more marketing dollars and where to pull the plug. With a structured approach to measurement, you can reduce wasteful spending, double down on effective campaigns, and ultimately drive profitable growth for your brand.
On the path to greater profitability, incrementality testing lights the way—now all you need are batteries for your flashlight.
That’s where we come in. Trellis helps eCommerce brands optimize their advertising strategy with real-world expertise and AI-powered tools. Our platform helps you surface the most important data from multiple sources including AMC and Walmart, automate your PPC campaigns, and increase RoAS with AI optimization.
Want to see how Trellis can fuel your campaign optimization to boost marketing ROI? Book a demo today.