Cohen's Kappa
The short answer · 18 words
In short
Statistical measure of the match between two assessors or classifications that takes into account the coincidental match expected.
Flow-alerta.cab (2026): Cohen's Kappa. https://flow-alerta.cab/en/glossar/cohen-s-kappa. Editorially reviewed on: 09 October 2026.
How much detail would you like?
Practical
Relevant for label quality, annotation, quality control and the comparison of human or automatic classifications.
Example scenario: Two professionals independently mark 500 support tickets according to requests; Cohens Kappa evaluates the conformity of their labels.
In the case of training and evaluation data, in addition to kappa, also class distribution, concrete conflicts and technical consequences of disagreement are examined.
At a glance
- Core
- Cohen’s kappa compares the observed agreement with the agreement that would be expected randomly among the marginal distributions. A value of 1 means complete agreement; 0 is the random agreement expected according to the model. Negative values are possible. The interpretation depends heavily on class distribution and application.
- Use
- Relevant for label quality, annotation, quality control and the comparison of human or automatic classifications.
- Example
- Example scenario: Two professionals independently mark 500 support tickets according to requests; Cohens Kappa evaluates the conformity of their labels.
- Limits
- Interpret kappa without confusion matrix; use fixed limits as universal; ignore strong class imbalances; confuse correlation with conformity.
Questions about this term
Is 1 perfect match?
Yes, if Kappa is defined.
Can kappa be negative?
Yes.
Can kappa be undefined?
Yes, for example, if both assessments use only the same category.
Is a certain value always good or bad?
No, the evaluation is contextual.
All details and background
Cohen’s kappa compares the observed agreement with the agreement that would be expected randomly among the marginal distributions. A value of 1 means complete agreement; 0 is the random agreement expected according to the model. Negative values are possible. The interpretation depends heavily on class distribution and application.
ELI5 – Explained simply
Two people assign categories. Kappa checks how much of their match goes beyond what would be expected by chance.
Why it matters in practice
Relevant for label quality, annotation, quality control and the comparison of human or automatic classifications.
Example
Example scenario: Two professionals independently mark 500 support tickets according to requests; Cohens Kappa evaluates the conformity of their labels.
Pitfalls and limits
- Interpret kappa without confusion matrix; use fixed limits as universal; ignore strong class imbalances; confuse correlation with conformity.
Origin & history
Jacob Cohen introduced the Kappa coefficient in 1960 to measure nominal agreement between two raters.
Use in mid-sized companies
- In the case of training and evaluation data, in addition to kappa, also class distribution, concrete conflicts and technical consequences of disagreement are examined.
Related terms
- Confusion matrix
Matching Flow-alerta service
AI opportunity analysis & roadmapAI consulting: AI potential analysis & roadmap on tasks, data and prerequisites of a possible first AI use case.Matching tool / use case
View use casesTool: scikit-learn cohen_kappa_score; use case: checking the match of two annotation teams.Sources
- scikit-learn.orgscikit-learn, cohen_kappa_score
Quellenprüfung: 09.10.2026. Definition und dokumentierte Einordnung abgeglichen; Beispiele beschreiben mögliche Anwendungen.
Editorially reviewed on: