HumanEval
Definition
HumanEval is a benchmark published in 2021 for functional correctness of generated Python code.
The dataset contains handwritten programming tasks with function signatures, docstrings, and tests. Results are often reported via pass@k.
Simply explained
Like a set of small programming tests where the generated code must pass the tests.
Why it matters in practice
Relevant for code generation benchmarks but insufficient for security, maintainability, or productivity in real repositories.
Example from business
A team compares several coding models on HumanEval and additionally on internal, isolated repository tasks.
Limits and pitfalls
- Running untrusted code without sandbox; equating benchmark score with secure production code; ignoring contamination or pass@k configuration.
Origin & history
Chen et al. introduced HumanEval in 2021 in the work on Codex.
Use in mid-sized companies
- Possible application: Part of a technical evaluation of coding assistants, supplemented by real internal tasks and code review criteria.
Frequently asked questions
What does HumanEval measure?
Functional correctness of small Python tasks. Does it measure security? No. Why sandbox? Model code can contain any dangerous operations.
Sources
- Evaluating Large Language Models Trained on Code · Chen et al. (2021), Evaluating Large Language Models Trained on Code
- OpenAI HumanEval Repository · OpenAI HumanEval Repository
Sources checked on 10.10.2026.
Editorially reviewed on: