How long to run an ad A/B test (and why stopping early fools you)
An ad A/B test should run until it reaches a sample size you chose before it started, not until the numbers look good. How to work that out, with worked examples, what Meta and Google Ads recommend, and a free calculator.
“How long should the test run?” has a real answer, and it is not a number of days. A test is long enough when it has seen enough people to tell the two versions apart. Days only come in when you divide that number of people by your traffic. Get the order right and the test tells you something. Get it wrong and you end up shipping whichever ad happened to be ahead on a lucky Tuesday.
The free A/B test calculator does the sums in this post in your browser. Here is how they work, so you know what to feed it.
Start from the smallest lift worth finding
Before the test, decide the smallest improvement you would act on. If a new hook lifted conversions by 3%, would you rebuild your ads around it? Probably not. By 20%? Probably yes. That smallest lift worth finding drives everything else, because small differences take far more data to see than big ones.
You also need three more numbers:
- Your current conversion rate. Take it from the ad, audience or landing page you are about to test, over a normal recent period.
- A confidence level. 95% is the usual choice. It sets how often a test with no real difference will still call a winner: about 5% of the time.
- Power. The chance of catching the lift if it is really there. Meta typically suggests 80% or higher for its own tests, and 80% is a common default.
Turn that into visitors per version
With those four numbers, the standard formula for comparing two conversion rates gives the visitors each version needs. The calculator shows the formula in full. Here is what it produces for a shop converting 2% of visitors, at 95% confidence and 80% power:
| Smallest lift worth finding | B’s rate | Visitors per version | Total |
|---|---|---|---|
| 30% | 2.6% | 9,798 | 19,596 |
| 20% | 2.4% | 21,109 | 42,218 |
| 10% | 2.2% | 80,682 | 161,364 |
Halving the lift you want to see roughly quadruples the traffic you need. That one fact explains most failed ad tests: someone tried to detect a small difference with a budget sized for a big one.
Raising the power has a cost too. The same 20% lift at 90% power needs 28,258 per version instead of 21,109.
Then, and only then, turn it into days
Divide the total by the visitors a day both versions will get together. At 3,000 visitors a day, the 20% test needs 42,218 ÷ 3,000 = 14.07, so 15 days. At 1,000 a day it needs 43 days. At 10,000 a day it needs 5.
Now sanity-check the answer against the platforms:
- Too short. Meta’s A/B testing best practices recommend at least 7 days and say shorter tests may produce inconclusive results. If your maths says 5 days, run 7 anyway. The same page says to run longer if your customers usually take more than 7 days to convert after seeing an ad.
- Too long. Meta caps A/B tests at 30 days. If your maths says 43, the test as designed cannot finish. Look for a bigger lift (test a bolder change), more traffic (a larger budget or a broader audience), or accept a lower confidence level and say so when you report the result.
- Google Ads. Its experiments help page recommends letting an experiment run for 2 to 3 weeks, and increasing budget or running longer if results are still undecided.
Why stopping early fools you
Here is the trap. You launch the test and check it every morning. On day four, B is ahead and the tool says significant. You stop, ship B, and move on.
The 95% confidence level only describes a test that is read once, at the planned sample size. If A and B truly convert the same, that single reading calls a false winner about 5% of the time. Checking every day is a different procedure. The two rates wander as visitors arrive, and every look is another chance for a chance gap to cross the line. Stop at the first crossing and you have picked the luckiest moment on purpose. The real false-positive rate ends up above 5%, and it climbs the more often you look.
The p-value on the screen cannot warn you, because it does not know how many times you checked. It reads exactly as if you had looked once.
Three rules keep a test honest:
- Write the sample size down before launch. Put it in the test name if you have to.
- Look, but do not stop. Checking that both versions are delivering and nothing is broken is fine. Ending the test because the number looks good is not.
- Read the result once, at the planned size. If it is not significant, that is an answer too: the difference, if any, is smaller than the lift you said was worth finding.
Read the result
When the test reaches its size, switch the calculator to Is my result significant? and enter visitors and conversions for both versions. Say A converted 200 of 10,000 visitors (2.00%) and B converted 245 of 10,000 (2.45%). The calculator gives z = 2.157 and a two-sided p-value of 0.031, below 0.05, so B wins at 95% confidence.
Then read the range, not just the verdict. The 95% interval for the difference runs from +0.04 to +0.86 percentage points, which as a lift on A is roughly +2% to +43%. B is better, but by how much is still wide open. If B costs more to produce, that range is what tells you whether it is worth it.
The platforms will not always agree with the calculator, and that is expected. Meta’s A/B results show a confidence percentage and treat 65% or higher as a winning result, a different measure from a p-value. Google Ads computes its intervals with jackknife resampling over 20 buckets per arm and shows an 80% interval by default. Use the platform’s verdict for the platform’s test, and the calculator to plan the size and sanity-check the maths.
Set the test up so the answer means something
- Change one thing. Meta’s best practices say results are more conclusive when the versions are identical except for the variable being tested.
- Split properly. Meta advises against testing by switching ad sets on and off, because audiences can overlap, and recommends the same budget for both versions. Use the platform’s own A/B test or experiment tool.
- Check the money, not just the rate. A version that wins on conversion rate can still lose money if it costs more per click. Run the winner through the ROAS calculator and compare it with your break-even ROAS before you scale it.
The short version
Pick the smallest lift worth finding, work out the visitors per version, divide by your daily traffic, keep the result between Meta’s 7 and 30 days, and do not stop early. The A/B test calculator does both halves. And if you would rather have the tests suggested for you, Hermoso is an AI marketing agent that runs your marketing on autopilot: it can read every ad platform you have connected and tell you what to scale, pause and test next, with each test’s hypothesis, the metric that decides it, a budget and a duration.
Questions people ask
How long should I run a Facebook ad A/B test?
Long enough to reach the sample size you worked out before starting, and at least 7 days, which is the minimum Meta recommends. Meta caps A/B tests at 30 days. At 3,000 visitors a day, a test on a 2% conversion rate looking for a 20% lift needs about 15 days.
How many conversions do I need for an A/B test?
It depends on your conversion rate and the lift you want to detect. At a 2% rate, finding a 20% lift at 95% confidence and 80% power takes 21,109 visitors per version, which is about 422 conversions in the original version.
Can I stop an A/B test early if it is already significant?
Not if you want the confidence level to hold. Checking often and stopping at the first significant reading gives chance more opportunities to cross the line, so the real false-positive rate ends up higher than the one you chose.
What if my test would need more than 30 days?
Meta A/B tests cannot run past 30 days, so change the design: test a bolder change so the lift you are looking for is bigger, add budget or broaden the audience for more traffic, or accept a lower confidence level and say so when you report the result.