Hypothesis Testing
Definition
A hypothesis test examines, based on data, how compatible an observation is with a formulated null hypothesis.
Typically, hypotheses, test statistics, significance level, and assumptions are predefined. A p-value is not the probability that the null hypothesis is true.
Simply explained
How to test whether an observed difference could plausibly be explained by chance – under clear model assumptions.
Why it matters in practice
Relevant for experiments, A/B tests, and data-driven decisions when uncertainty should be transparently quantified.
Example from business
A company tests whether a new landing page measurably changes conversion in a controlled experiment.
Limits and pitfalls
- Misinterpreting p-values; many tests without correction; ignoring effect size; post hoc hypothesis shopping; claiming causality without experiment.
Origin & history
Statistical hypothesis tests work with null and alternative hypotheses and controlled error risks. The ASA explains limits of p-value interpretation.
Use in mid-sized companies
- Possible applications: A/B tests, process improvements, quality comparisons, and model metrics.
Frequently asked questions
Does p<0.05 mean a large effect?
No. Does a non-significant test prove equality? No, power and uncertainty must be considered.
Sources
- ASA Statement on Statistical Significance and P-Values · ASA Statement on Statistical Significance and P-Values
- What are statistical tests · NIST, What are statistical tests
Source check: 10./11.10.2026.
Editorially reviewed on: