Statistical Significance
Definition
Statistical significance indicates a test result unusual under the null hypothesis, not automatically an important effect.
Statistical significance means that a test result is sufficiently unusual under a specified null hypothesis, not that the effect is large or practically important.
Simply explained
You test an initial assumption with data. An unusual result may argue against this assumption but alone does not indicate how large or useful the difference is.
Why it matters in practice
The term helps to select, develop, evaluate AI systems in a well-founded manner and realistically assess their technical risks.
Example from business
An A/B test reports p-value, confidence interval, and absolute effect size instead of just "significant."
Limits and pitfalls
- P-hacking, multiple testing, and dichotomous interpretation of p-values create false confidence.
Origin & history
Significance tests developed in the early 20th century; misinterpretations led to current recommendations for transparent reporting of effects and uncertainties.
Use in mid-sized companies
- Possible application: An A/B test reports p-value, confidence interval, and absolute effect size instead of just "significant."
Frequently asked questions
What does Statistical Significance mean?
Statistical significance means that a test result is sufficiently unusual under a specified null hypothesis, not that the effect is large or practically important. What should you watch out for? P-hacking, multiple testing, and dichotomous interpretation of p-values create false confidence.
Sources
- Statement on Statistical Significance and P-Values · ASA, Statement on Statistical Significance and P-Values
Source checked on 10.10.2026. Editorial application examples are hypothetical; no evidence of real customer results.
Editorially reviewed on: