A/B Test Duration Calculator -- Sample Size Tool

Estimate A/B Test Duration

This A/B test duration calculator estimates the sample size and test length needed to reach statistical significance. Enter your traffic, current conversion rate, and the smallest improvement you want to detect.

Worked Examples

Three real-world scenarios showing how traffic, baseline rate, and MDE interact to determine test duration.

How A/B Test Sample Size Is Calculated

The required sample size per variation is determined by four inputs:

• Baseline conversion rate (p₁): Your current conversion rate before the test
• Minimum Detectable Effect (MDE): The smallest relative improvement worth detecting. A 10% MDE on a 5% baseline means detecting the difference between 5.0% and 5.5%
• Confidence level (1 − α): How certain you need to be that a detected difference is real, not random noise. 95% is standard (5% false positive risk)
• Statistical power (1 − β): The probability of detecting a real effect when it exists. 80% is standard (20% false negative risk)

The formula:

n = 2 × [(Z_α/2 + Z_β)² × (p₁(1−p₁) + p₂(1−p₂))] / δ²

Where:

• n = required sample size per variation
• Z_α/2 = Z-score for the confidence level (1.96 for 95%)
• Z_β = Z-score for statistical power (0.84 for 80%)
• p₁ = baseline conversion rate
• p₂ = expected new conversion rate (p₁ × (1 + MDE))
• δ = absolute difference (p₂ − p₁)

Test duration is then: sample size ÷ daily visitors per variation (rounded up, minimum 14 days).

What Changes Test Duration Most?

These factors have the biggest impact on how long your test needs to run:

Key insight: MDE has the largest impact because it's squared in the denominator. Relaxing your MDE from 5% to 10% cuts sample size by roughly 75%. If a test requires more time than you can afford, raising MDE is usually the most practical lever.

Common A/B Testing Mistakes

• Running tests too short: A test that runs for only a few days may hit "significance" by chance. Without 1–2 full business cycles (weekday + weekend), results are unreliable.
• Testing too many variations: Each additional variation requires more total traffic. A test with 4 variations needs roughly 3× the traffic of a standard A/B test to maintain statistical power.
• Changing the test mid-run: Modifying the control or variation during the test invalidates the results. If you need to make changes, restart the test with a new sample.
• Ignoring novelty effect: Users may interact differently with new designs simply because they are new. Run tests long enough (3–4 weeks) for the novelty to wear off.
• Not segmenting results: A test might show no overall lift but a significant lift for mobile users. If you plan to analyze segments, account for this in your sample size (more visitors needed).
• Using the wrong metric: If your goal is revenue, optimize for revenue per visitor — not just conversion rate. A variation might convert more visitors but at a lower order value.

A/B Testing Best Practices

• Calculate sample size before starting. Use this calculator. Write down the required visitors and commit to the duration.
• Run for at least 2 full weeks to capture weekday and weekend traffic patterns, regardless of what the math says.
• Test one change at a time for clear causation. If you change the headline, button, and layout simultaneously, you cannot attribute the result to any single element.
• Use a 50/50 traffic split for maximum statistical efficiency. Uneven splits (90/10) require dramatically more visitors.
• Avoid testing during unusual periods: holidays, sales events, product launches, or marketing campaigns can skew results.
• Document everything: hypothesis, expected MDE, sample size, start date, end date, and decision. This builds institutional knowledge and prevents revisiting settled questions.
• Accept null results gracefully. Most tests do not produce a winner. A null result is still valuable — it tells you the change does not meaningfully impact the metric.

When You Don't Have Enough Traffic

Many sites cannot run statistically rigorous A/B tests. If your calculator shows a test would take 6+ months, consider these alternatives:

• Increase MDE. If you can only detect 30%+ lifts, focus on bold changes (complete redesigns, new value propositions) rather than minor tweaks.
• Use a higher-traffic page. Test on your homepage or landing page instead of a low-traffic settings page.
• Combine traffic from multiple pages. Some tools support site-wide A/B tests that aggregate traffic across pages.
• Run pre/post tests. Compare metrics before and after a change (less rigorous, but better than guessing).
• Use qualitative methods. User interviews, session recordings, and usability tests can reveal improvements without statistical testing.

Frequently Asked Questions

How long should I run an A/B test?

It depends on your daily traffic, baseline conversion rate, and the smallest improvement you want to detect. Most tests need 2–6 weeks. Always run for at least 14 days to capture weekly patterns. Use this calculator to get an exact estimate for your numbers.

How do I calculate A/B test sample size?

Use the formula: n = 2 × [(Z_α/2 + Z_β)² × (p₁(1−p₁) + p₂(1−p₂))] / δ². At 95% confidence and 80% power, with a 5% baseline and 10% relative MDE, you need approximately 31,000 visitors per variation. This calculator handles the math for you.

What is Minimum Detectable Effect (MDE)?

MDE is the smallest relative improvement in conversion rate your test is designed to detect. A 10% MDE on a 5% baseline means detecting a lift from 5.0% to 5.5%. Smaller MDEs require larger sample sizes. Most teams use 5–20% MDE depending on traffic and expected impact.

What confidence level should I use?

95% is the industry standard (5% false positive risk). Use 90% for exploratory tests where speed matters. Use 99% for high-stakes decisions like pricing changes. Lower confidence means shorter tests but higher risk of declaring false winners.

What is statistical power in A/B testing?

Statistical power is the probability of detecting a real effect. At 80% power (standard), there is a 20% chance of missing a real improvement. Increasing power to 90% adds roughly 30% more required visitors but reduces false negatives.

Can I stop an A/B test early if results look significant?

No. "Peeking" — stopping early when results look good — inflates your false positive rate from 5% to as high as 25–50%. Decide your sample size before starting and commit to it. If you need flexibility to stop early, use sequential testing methods specifically designed for it.

Why is 14 days the minimum test duration?

User behavior differs between weekdays and weekends. Conversion rates, traffic volume, and visitor intent can shift significantly. A test running Tuesday through Friday misses weekend patterns entirely. Two full weeks captures at least two of each day.

What happens if my test runs too short?

An underpowered test produces unreliable results. You might declare a winner that is actually no better (false positive), or miss a real improvement (false negative). Both outcomes waste time and can lead to worse decisions than not testing at all.

How does baseline conversion rate affect duration?

Lower baselines require more visitors. A site converting at 1% needs roughly 5× more visitors than a site converting at 5% to detect the same relative improvement. Lower rates have higher variance relative to the effect size.

What if my test shows no significant result?

A null result means the change did not produce a detectable effect at your MDE level. This is valuable information — it means the change is unlikely to have a meaningful impact. Do not re-run the same test hoping for a different result. Move on to a higher-impact hypothesis.

How many variations should I test?

Start with two (control + one variation). Each additional variation multiplies the required total traffic. A test with 4 variations needs roughly 3× the traffic of a standard A/B test. Only add variations when you have enough traffic to support them.

Does this calculator send my data to a server?

No. All calculations run entirely in your browser using JavaScript. Nothing is transmitted, stored, or logged.

Related Tools

• Conversion Rate Calculator — calculate conversion rates from visitors and conversions
• Conversion Funnel Calculator — analyze drop-off rates across funnel stages
• Bounce Rate Calculator — calculate and analyze bounce rates
• Session Duration Calculator — measure average session length
• Confidence Interval Calculator — calculate confidence intervals for proportions and means

Learn More

For a deeper explanation of sample size calculations, statistical significance, and practical tips for running your first A/B test, read our guide: How Long Should You Run an A/B Test? A Practical Guide to Sample Size and Duration.

Privacy & Limitations

• Client-side only. No data is sent to any server. No cookies, no tracking.
• Two-tailed test. This calculator assumes a two-sided hypothesis test (detecting both positive and negative effects).
• Two variations only. The formula assumes a standard A/B test with one control and one variation. For multivariate tests, sample size requirements increase.
• Assumes equal traffic split. An uneven split (e.g., 90/10) requires a larger total sample size than a 50/50 split.
• Does not account for multiple comparisons. If you plan to analyze multiple metrics or segments, consider applying a Bonferroni correction or using a lower significance threshold.

A/B Test Duration Estimator FAQ