Inter-Annotator Agreement
Definition
Inter-Annotator Agreement measures how much multiple annotators agree on the same data.
Depending on data type and number of raters, metrics like Cohen’s Kappa, Fleiss’ Kappa, or Krippendorff’s Alpha are used. High agreement does not automatically mean labels are factually correct.
Simply explained
How to check whether multiple people rate the same examples similarly according to the same rules.
Why it matters in practice
Relevant for training and evaluation data because unclear guidelines often directly affect label quality and model evaluation.
Example from business
Two experts classify the same complaints; significant discrepancies lead to a revision of the labeling guideline.
Limits and pitfalls
- Overinterpret percentage agreement without chance correction; choose an inappropriate metric; overlook systematic shared bias.
Origin & history
Cohen published Kappa in 1960 for agreement between two raters on categorical judgments; further measures followed for other designs.
Use in mid-sized companies
- Possible applications: quality assurance in annotation, human evaluation, and expert review processes.
Frequently asked questions
Does high IAA mean correct labels?
Not necessarily. Which metric is correct? It depends on the scale, number of annotators, and missing values.
Sources
- A Coefficient of Agreement for Nominal Scales · Cohen (1960), A Coefficient of Agreement for Nominal Scales
Sources checked on 10.10.2026.
Editorially reviewed on: