MBPP
Definition
MBPP checks whether models solve small Python tasks.
The dataset contains short natural language programming tasks, reference programs, and tests. Generated code is verified based on its function.
Simply explained
The model receives a small programming task and must pass appropriate tests.
Why it matters in practice
MBPP makes basic code generation comparable between models.
Example from business
Hypothetical: A team compares models on list and string tasks.
Limits and pitfalls
- Few tests do not cover all errors. Known tasks may be included in training data.
Origin & history
MBPP was introduced in the paper Program Synthesis with Large Language Models 2021.
Use in mid-sized companies
- SMEs can use it as an additional signal for selecting a programming assistant.
Frequently asked questions
Does a high score prove production-ready code?
No, security and integration require separate testing.
Sources
- Program Synthesis with Large Language Models — · Austin et al.: Program Synthesis with Large Language Models —
Editorially reviewed on: