MMLU
Definition
MMLU evaluates knowledge and problem-solving across many disciplines.
The benchmark includes multiple-choice questions from various disciplines and difficulty levels. Evaluation measures the proportion of correct answers according to a defined procedure.
Simply explained
A broad knowledge test for language models.
Why it matters in practice
It provides a comparative signal for cross-disciplinary model capabilities.
Example from business
Hypothetical example: A team considers, besides the overall score, the domain-appropriate subfields.
Limits and pitfalls
- Guessing, data overlap, and prompt choice influence scores. Practical tasks may differ significantly.
Origin & history
Hendrycks and co-authors published MMLU in 2020.
Use in mid-sized companies
- SMEs should use it in addition to their own examples.
Frequently asked questions
Does MMLU measure the quality of long advisory conversations?
No, it evaluates a fixed task format.
Sources
- Measuring Massive Multitask Language Understanding — · Hendrycks et al.: Measuring Massive Multitask Language Understanding —
Editorially reviewed on: