A/B Test
Definition
An A/B test compares randomly assigned variants based on predefined success metrics.
An A/B test randomly assigns users to variants and compares predefined metrics to estimate causal effects of the change.
Simply explained
Two groups receive different versions; afterwards, it is compared which performs better on the predefined goal.
Why it matters in practice
The term helps to make digital offerings understandable, accessible, and aligned with users' needs.
Example from business
A shop tests two checkout forms with a predefined conversion metric and duration.
Limits and pitfalls
- Early stopping, many unplanned metrics, contamination, and lack of practical relevance distort results.
Origin & history
Controlled experiments originate from statistics and medicine; online services scaled them from the 2000s for product decisions.
Use in mid-sized companies
- Possible application: A shop tests two checkout forms with a predefined conversion metric and duration.
Frequently asked questions
What does A/B test mean?
An A/B test randomly assigns users to variants and compares predefined metrics to estimate causal effects of the change. What should be considered? Early stopping, many unplanned metrics, contamination, and lack of practical relevance distort results.
Sources
- Trustworthy Online Controlled Experiments · Kohavi et al., Trustworthy Online Controlled Experiments
Source checked on 10.10.2026. Editorial application examples are hypothetical; no evidence of real customer results.
Editorially reviewed on: