A poorly run A/B test is worse than no test at all: it produces certainty about something false. This guide covers the full cycle, hypothesis, sample, duration, reading significance, and the mistakes that invalidate experiments, so every test returns a decision you can trust.
What A/B testing is
A/B testing is a controlled experiment that splits traffic between two versions (control and variant) to measure which one produces the better result on a metric defined before it starts. It is the central tool of any CRO process.
The split needs to be random and simultaneous. Comparing "this week to last week" is not an A/B test; it is a comparison contaminated by seasonality, media, and behavior.
How to write the hypothesis
A hypothesis without evidence is a guess. Before testing, collect the signal: session recordings, scroll maps, form abandonment rate, or objections reported by sales.
Sample, duration, and statistical power
- Define the primary metric, just one. Secondary metrics act as guardrails.
- Estimate the minimum detectable effect (MDE) that justifies the change, for example +10%.
- Calculate the sample size needed per variant before starting.
- Run it through full behavioral cycles, usually 2 to 4 weeks.
- Do not stop at the first spike you see: an early swing is noise.
| Baseline rate | Minimum effect | Order of magnitude per variant |
|---|---|---|
| 2% | +20% relative | Tens of thousands of visitors |
| 5% | +20% relative | Thousands to tens of thousands |
| 10% | +20% relative | A few thousand |
| 20% | +20% relative | Hundreds to thousands |
If your volume cannot reach the sample, change strategy: test bigger changes (offer, value proposition, audience), which produce larger effects and need less traffic to show up.
How to read the result
Statistical significance
The market standard is 95% confidence (p-value below 0.05). That means roughly a 5% chance the observed result is due to chance. Report the confidence interval too: a gain of "+12% between +1% and +25%" is very different from "+12% between -8% and +32%."
Practical significance
Not every statistical gain is worth it. If the full implementation is expensive and the projected gain is marginal, the correct outcome is to archive it with the learning logged.
7 mistakes that invalidate the test
Check these before declaring a winner
- Ending the test as soon as a difference appears (peeking).
- Running it for less than one full weekly cycle.
- Changing the variant or the media during the experiment.
- Testing on a sample too small for the effect you are after.
- Measuring clicks when the real goal is a qualified sale.
- Ignoring the separate performance of mobile and desktop.
- Not logging the learning, and repeating the same test months later.
Conclusion
Testing well is a discipline of intellectual honesty: set the criterion first, respect the time needed, and accept the result. Done this way, every experiment improves today's page and tomorrow's decision about new landing pages.
Frequently asked questions about a/b testing
What is A/B testing?
It is a controlled experiment that splits traffic between two or more versions of a page or element to measure which produces the better result on a metric defined before the test.
How long should an A/B test run?
At least one full behavioral cycle, usually 2 to 4 weeks, and until the planned sample size is reached. Stopping earlier tends to produce false positives.
What is statistical significance in an A/B test?
It is the probability that the observed result is not due to chance. The market standard is 95% confidence, which corresponds to a p-value below 0.05.
Can I test more than one change at the same time?
You can, but then the test tells you whether the combination is better, not which element caused the gain. To isolate causes, test one variable at a time or use a multivariate design with enough volume.
What should I do when a test ties?
A tie is also a learning: it means the variable tested is not the lever. Log the result and move to a higher-impact hypothesis, such as offer, audience, or value proposition.
About the author
Kauára Bertoluci
Partner and Head of Strategy & Growth, KCLOCK
Marketing, tracking, and CRO strategist. Leads positioning, funnels, paid media, and measurement across KCLOCK's projects.
- #testing
- #data
- #cro
Want to apply this to your funnel with people who do it every day?
KCLOCK builds high-converting funnels, landing pages, websites, and e-commerce stores. We diagnose your current funnel and show you where the revenue you're leaving on the table is hiding.
