A/A Test
An A/A test splits traffic between two identical versions of a page. It is essential quality assurance before running real experiments.
Written by Sam Flynn
TL;DR
An A/A test splits traffic between two identical versions of a page to verify that your testing tool is working correctly and that your baseline conversion rates are stable. It is essential quality assurance before running real experiments, false winners in A/A tests reveal flawed measurement infrastructure.
What Is an A/A Test?
An A/A test is a controlled experiment in which both the control and the variant are exactly the same. No design changes, no copy changes, visitors are split into two groups that see identical experiences. If your testing platform is functioning correctly, neither version should produce a statistically significant winner.
The primary use case is validation. Before trusting A/B test results, CRO teams run A/A tests to confirm that the experiment engine assigns users randomly, applies changes consistently, and tracks conversions without bias. If an A/A test reports a winner, something in the measurement stack is broken.
A/A tests are also used to estimate natural conversion rate variance over time. By observing how much the rate fluctuates between identical groups, teams calibrate their minimum detectable effect (MDE) thresholds and avoid chasing noise in future experiments.
How It Works
| Step | Action | Why It Matters |
|---|---|---|
| 1 | Set up two identical page variants in your testing tool | Baseline, no changes introduced |
| 2 | Split traffic 50/50 | Ensures equal exposure between groups |
| 3 | Run for full business cycles (≥2 weeks) | Captures day-of-week and seasonal variance |
| 4 | Check for statistical significance | A winner = tool or tracking problem |
| 5 | Review sample ratio mismatch (SRM) | Unequal group sizes signal assignment bugs |
Benchmarks
| Metric | Expected Outcome | Red Flag |
|---|---|---|
| Statistical significance | p > 0.05 (no winner) | p < 0.05 suggests tool error |
| Sample ratio mismatch | Groups within 1% of each other | >2% split deviation |
| Conversion rate delta | <0.5% between groups | >2% persistent delta |
| False positive rate | ~5% at 95% confidence | >10% across repeated tests |
Best Practices
| Practice | Explanation |
|---|---|
| Run before every new experiment programme | One-time validation is not enough, re-validate after tool updates |
| Use full traffic, not a subset | Small samples inflate variance and produce misleading results |
| Run for at least two full weeks | Captures weekly patterns in user behaviour |
| Check SRM first | Sample ratio mismatch is a more reliable diagnostic than p-value alone |
| Document results | Build a historical baseline of natural conversion variance |
Common Mistakes
- Skipping the A/A test entirely and trusting tool defaults
- Running too short a duration, producing high variance readings
- Treating a failed A/A test as a conversion insight rather than an infrastructure problem
- Not checking for SRM, unequal splits invalidate all downstream analysis
- Running A/A tests only once during initial setup and never repeating after platform changes
Key Takeaways
- An A/A test runs identical variants to validate measurement infrastructure, not to learn about users
- A statistically significant result in an A/A test means your testing tool or tracking has a bug
- Always check for sample ratio mismatch, uneven group sizes invalidate results
- Run A/A tests for at least two weeks to capture natural conversion variance
- A/A testing is the quality control step that makes all subsequent A/B test results trustworthy
- Document A/A baselines to set realistic MDE thresholds for future experiments
