MBPP — Results
| Rank | Model | Score |
|---|---|---|
| 1 | llama-3-3-nemotron-super-49b-v1 | 0.913 |
| 2 | qwen2-5-coder-32b-instruct | 0.902 |
| 3 | qwen2-5-72b-instruct | 0.882 |
| 4 | qwen2-5-32b-instruct | 0.84 |
| 5 | qwen2-5-vl-32b-instruct | 0.84 |
| 6 | qwen2-5-coder-7b-instruct | 0.835 |
| 7 | qwen2-5-14b-instruct | 0.82 |
| 8 | qwen3-235b-a22b | 0.814 |
| 9 | phi-3-5-moe-instruct | 0.808 |
| 10 | qwen2-72b-instruct | 0.802 |
| 11 | qwen2-5-7b-instruct | 0.792 |
| 12 | llama-4-maverick | 0.776 |
| 13 | mistral-small-3-1-24b-instruct | 0.747 |
| 14 | qwen2-5-omni-7b | 0.732 |
| 15 | mistral-small-3-24b-base | 0.696 |
| 16 | phi-3-5-mini-instruct | 0.696 |
| 17 | llama-4-scout | 0.678 |
| 18 | qwen2-7b-instruct | 0.672 |
MBPP
1 phaseActive
974 crowd-sourced Python programming problems for entry-level coders. Tests code generation from natural language descriptions with automated test cases. Metric: pass@1.
Overview
MBPP
Quick answer: MBPP (Mostly Basic Python Problems) is a crowd-sourced Python programming benchmark from Google Research (Austin et al., 2021). It contains 974 programming problems designed to be solvable by entry-level programmers, each with a task description, a canonical solution, and 3 automated test cases. Models are scored on pass@1 — whether the generated code passes all tests on the first attempt.
At a Glance
What it tests: Code generation from natural language problem descriptions — the ability to write correct Python functions that pass automated test suites. Problems cover standard programming fundamentals: string manipulation, list operations, arithmetic, data structures, and standard library usage.
Why it matters: MBPP is one of the most widely used code generation benchmarks alongside HumanEval. Its 974 problems provide better statistical reliability than HumanEval's 164. It complements HumanEval by covering a broader, crowd-sourced problem distribution.
Known limitations: MBPP is heavily saturated — frontier models regularly score above 80–90%. The 3 test cases per problem are minimal, meaning some incorrect solutions may pass. MBPP+ variants add more test cases for stricter evaluation.
What MBPP Measures
MBPP covers Python programming fundamentals at entry to intermediate level. Unlike competitive programming benchmarks (CodeForces, AIME for code), MBPP problems are straightforward: given a function description, write a Python function that passes the provided tests.
Problem domains include: string operations, list/dictionary manipulation, mathematical computations, date/time operations, sorting and searching, pattern matching, and basic algorithms.
Evaluation variants:
- MBPP (sanitized): A curated 374-problem subset with higher-quality descriptions
- MBPP+: Enhanced version with 35x more test cases per problem (using EvalPlus framework)
Benchmark Specifications
| Field | Value |
|---|---|
| Total problems | 974 |
| Language | Python |
| Primary metric | pass@1 |
| Test cases | 3 per problem (original) / 35x more in MBPP+ |
| Created by | Austin et al. (Google Research) |
| Paper | arXiv:2108.07732 (Aug 2021) |
| Saturation | High — frontier models 80–90%+ |
Frequently Asked Questions
How does MBPP compare to HumanEval? MBPP has 974 problems vs HumanEval's 164, providing better statistical reliability. HumanEval problems are generally harder and more focused on algorithmic thinking; MBPP skews toward practical, everyday coding tasks. Most labs report both.
What is a good MBPP score? Frontier models consistently score 80–90%+ pass@1 on MBPP. Models from 2023 typically scored 60–75%; the benchmark has become increasingly saturated.
What is MBPP+ and how does it differ? MBPP+ uses the EvalPlus framework to generate approximately 35x more test cases per problem, catching edge cases that the original 3 tests miss. Scores on MBPP+ are typically 5–15 points lower than standard MBPP.