Benchmark
The short answer · 17 words
In short
Standardised task, test collection or measurement method used to evaluate systems under the most comparable conditions possible.
Flow-alerta.cab (2026): Benchmark. https://flow-alerta.cab/en/glossar/benchmark. Editorially reviewed on: 09 October 2026.
How much detail would you like?
Practical
Helps with model comparison and quality assurance, but can only partially reflect real company requirements. Own tests are often necessary.
Example scenario: Three language models are compared on the same 200 support questions for accuracy, source reference, latency and cost.
Use public benchmarks only as a pre-selection and supplement them with their own test set of real tasks, data and quality criteria.
At a glance
- Core
- A benchmark specifies data, tasks, metrics, and often an evaluation protocol. In AI and machine learning, models are compared on defined capabilities. A benchmark only measures what its tasks and metrics actually cover.
- Use
- Helps with model comparison and quality assurance, but can only partially reflect real company requirements. Own tests are often necessary.
- Example
- Example scenario: Three language models are compared on the same 200 support questions for accuracy, source reference, latency and cost.
- Limits
- Interpreting leaderboard value as general intelligence; comparing different test conditions; ignoring benchmark contamination; looking at just one metric.
In context
Benchmark Contamination
Contamination of model evaluation when benchmark tasks or closely related information were already available during training, tuning or development.
BIG-bench
Collaboratively developed benchmark collection with more than 200 tasks to study different capabilities and limitations of large language models.
Questions about this term
Is the best benchmark score automatically the best model for a company?
No.
What does a fair comparison need?
Same tasks, metrics and conditions.
Can benchmarks become obsolete?
Yes.
All details and background
A benchmark specifies data, tasks, metrics, and often an evaluation protocol. In AI and machine learning, models are compared on defined capabilities. A benchmark only measures what its tasks and metrics actually cover.
ELI5 – Explained simply
Several systems get the same test. The results can be compared as long as the test and rules are really the same.
Why it matters in practice
Helps with model comparison and quality assurance, but can only partially reflect real company requirements. Own tests are often necessary.
Example
Example scenario: Three language models are compared on the same 200 support questions for accuracy, source reference, latency and cost.
Pitfalls and limits
- Interpreting leaderboard value as general intelligence; comparing different test conditions; ignoring benchmark contamination; looking at just one metric.
Origin & history
Standardized benchmarks have been widely used in computer science and machine learning for decades. With large language models, numerous domain-specific and broad benchmark collections were created.
Use in mid-sized companies
- Use public benchmarks only as a pre-selection and supplement them with their own test set of real tasks, data and quality criteria.
Related terms
Matching Flow-alerta service
AI opportunity analysis & roadmapAI consulting: AI potential analysis & roadmap on tasks, data and prerequisites of a possible first AI use case.Matching tool / use case
View use casesUse Case: standardized model comparison plus company’s own evaluation set.Sources
- developers.google.comGoogle for Developers, Machine Learning Glossary
- mlcommons.orgMLCommons Benchmarks
Quellenprüfung: 09.10.2026. Definition und dokumentierte Einordnung abgeglichen; Beispiele beschreiben mögliche Anwendungen.
Editorially reviewed on: