Before you start
- A/B testing is currently in development and not yet available to all customers. If you're interested in early access, please reach out to your Account Manager or Customer Support Team.
A/B testing is a method of comparing two variations of an ad set within the same campaign to determine which one drives stronger performance. Running an A/B test can answer questions such as:
- Does offering 15% off to first-time site visitors increase purchases without killing margin?
- Do dynamic product ads perform better than a static brand creative for our catalog?
- Does showing a personalized “recently viewed” message convert more than a general promotion?
A/B testing allows you to optimize your ad creatives based on performance data instead of untested assumptions.
How A/B Tests work in this beta?
During this test you will create and upload two ad variants (A and B) to the AdRoll platform. AdRoll shows Variant A to one group of unique users and Variant B to another (a 50/50 split). AdRoll measures performance using click-through rate and applies statistical testing to determine whether the observed difference reflects a true lift or normal variation. The tests are spend-balanced automatically. You don’t need to micromanage budgets or run messy “two-campaign” comparisons – AdRoll keeps the experiment fair for you.
Requirements
When setting up an A/B test, both variants must use the same ad sizes to keep the comparison fair. We recommend uploading as many sizes as possible – more sizes unlock more placements and usually drive more delivery and clicks, helping you reach results faster.
Supported sizes include:
- 970x250
- 728x90
- 160x600
- 300x600
- 300x250
- 320x50
A/B test ads must run inside one or more campaigns. The test runs only while the campaign is active (and pauses when it’s paused), and it won’t automatically stop when a winner is found.
A/B testing needs enough data to separate a real lift from normal randomness, so you’ll likely need hundreds – sometimes thousands – of clicks before a statistical winner can be determined. How quickly you get there depends on:
- Traffic volume
- Targeting
- How different your variants are (bigger differences are easier to detect; small tweaks usually take more clicks)
Create an A/B Test
Go to A/B Testing
In the left navigation, expand Advertising and select A/B Creative Tests. On the A/B Creative Tests page, click Create A/B Test.
If you have no previous experiments, it will show you the following page asking you to create an A/B Test. Otherwise you will see a list of previous experiments.
Fill in Test Details
In the Create A/B Test window:
- Test name: choose something descriptive (you may have many experiments over time).
- Destination URL: set where clicks should land
Add ads for each variant
Upload ads into Variant A and Variant B. You can upload multiple ad formats (e.g., sizes), but:
- You must upload at least one format, and
- You should upload the same set of formats for both variants (so the comparison is fair).
Create the experiment
When both variants are filled out, click Create.
After you create it
A/B Test Dashboard
You’ll now see the experiment listed on the A/B Tests homepage. New experiments won’t show results immediately.
There are several possible statuses:
- Collecting Data: The test is live and still gathering enough impressions/clicks to make a reliable comparison. AdRoll does not yet have enough evidence to confidently call a winner.
- Too Close to Call: The test has collected data, but the results are too similar (or too noisy) to confidently say one variant is better. In other words, any observed difference could still plausibly be due to chance.
- Winner Determined: One variant is currently outperforming the other strongly enough that the statistical test indicates the result is unlikely to be random chance. This variant is the “winner” based on the data collected so far.
Note
- Collecting Data is the starting state. Once there’s enough traffic to evaluate results, the test will move to Too Close to Call or Winner Determined.
- If results are very close, especially early on, additional data can change the outcome, and the status may update between Too Close to Call and Winner Determined as the test stabilizes.
- Use the status as a decision aid, and review results alongside your business context (seasonality, promotions, audience changes, creative fatigue) before making a final call.
A/B Tests in Ad Library
You can also find the A/B test ads in the Advertising > Ad Library, marked with an A/B Test label.
Start running the A/B Test
Creating the A/B test does not start it automatically. To run the test, you must add the A/B test ad to a campaign. The test starts and stops based on whether the campaign is active.
To add it to a campaign, select the A/B test ad from Ad Library when choosing ads for your campaign.
Interpreting Your Results
Once your A/B test is live, our system begins analyzing the performance of your variants daily. Because we use statistical hypothesis testing to ensure accuracy, your test will fall into one of three states.
A Clear Winner Identified
What it means: One variant is performing significantly better than the other, and we have enough data to be confident that this isn't just a fluke.
- Action: You’ll see a "Winner" badge on the top-performing variant. You can now confidently shift your full budget to this creative or use its design elements as a blueprint for future campaigns.
In Progress: More Data Needed
What it means: The test is still running, and the performance gap between Variant A and Variant B isn't wide enough yet to declare a winner with statistical certainty.
- Action: Patience is key! We recommend letting the test run until it reaches at least 1,500 clicks.
No Significant Difference (Tie)
What it means: We have collected a large volume of data, but both variants are performing almost identically. Statistically, we can’t distinguish between the two.
- Action: This is still a win! It tells you that the specific change you tested (e.g., a blue button vs. a green button) doesn't significantly impact your audience's behavior. You are free to use either version or start a new test with a more distinct variable, like a completely different offer or imagery.
Understanding the Cumulative Results
In addition to the high-level conclusion, the reporting shows detailed results on the test.
| Category | Description |
|---|---|
| Test Duration | The Test Duration table shows the first and last day of the A/B tests and the total number of elapsed days. |
| Performance Metrics | The Performance Metrics table shows, for each variant, the cumulative number of impressions served and clicks received. We also report the click probability as a percent chance. Note that for the statistical analysis we only count unique cookies each day so these numbers may differ somewhat from other campaign reporting. |
| Statistical Confidence | The statistical reporting shows, for each ad variant, the probability that it is, in fact, the best. This is the key result. These numbers are computed from a combination of a.) the difference in the click probabilities and b.) total amount of data obtained so far. Generally one can call the test complete if one variant has a >95% chance to be the best. |
| Graphical Results | The tables above report the cumulative results, based on aggregated data from all days. The graphs show the evolution of these numbers as the data has been acquired. Graphs show the number of impressions and clicks received each day (the Daily Impressions/Clicks graphs) and the Cumulative number of impressions/clicks received up to the given date (the Cumulative Impressions/Clicks graphs). It’s typical to see daily ups and downs in the number of clicks and impressions, and the cumulative number of impressions and clicks should march steadily upward. Finally, the graph of click probability shows the click probability based on the cumulative impressions/clicks up to that point in the experiment. The shaded band shows the statistical uncertainty in the click probability. This chart can be read by looking for not just a higher click rate, but uncertainty bands that are well separated from each other. Generally, the bands will overlap substantially at the start of the experiment (when we do not know which variant is winning) and tend to separate by the last day. |
FAQs and Limitations
A/B Testing is currently in beta. More features are coming. To help prioritize improvements, contact your Account Manager or Support and share your use case.
Can I A/B test ads outside of the main six (970x250, 728x90, 160x600, 300x600, 300x250)?
Not during this Beta test.
Can I A/B test my campaign setups?
Not during this Beta test.
Can I test which version of my ad is better at getting conversions?
No. Conversions are harder to get than clicks and often come substantially after the ad impression. This makes it harder to get the required data volume for statistical significance and complicates the analysis. We are sticking to clicks for now.
Can I A/B test landing pages?
Not during this Beta.
Can I A/B test existing ads?
There is no way to choose existing ads. You may certainly re-upload the creatives.
How much budget do you recommend to run an A/B test?
Your budget is dependent on the cost of your CPC. Here is a formula you can use to estimate the budget you may need to allocate:
- Step 1 - Determine impressions needed: =1000 clicks / [Avg CTR]
- Step 2 - Determine minimum budget recommendation: = (Impressions needed * [Avg CPM]) / 1000