# MBPP+ ![Category](https://img.shields.io/badge/Category-Coding-1f6feb) ![Metric](https://img.shields.io/badge/Metric-Pass%40k-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-399-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-high-d29922) ![Created](https://img.shields.io/badge/Created-May%202023-555) [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/abs/2305.01210) [![GitHub](https://img.shields.io/badge/GitHub-evalplus%2Fevalplus-181717?logo=github&logoColor=white)](https://github.com/evalplus/evalplus) > **Quick answer:** MBPP+ is the EvalPlus framework's rigorously re-tested and sanitized version of MBPP (Mostly Basic Python Problems), narrowing the original dataset to 399 well-formed entry-level problems and dramatically expanding the test cases per problem so that superficially-passing but subtly-wrong code can no longer hide. ## At a Glance **What it tests:** Whether LLM-generated Python solutions to entry-level programming problems are genuinely correct across a much wider range of inputs than MBPP's original, sparser test suites checked for. **Why it matters:** Like HumanEval+, MBPP+ exposed that a meaningful share of previously "passing" model solutions on MBPP were subtly wrong, and that stricter testing can reorder which models actually generate the most reliable code. **Known limitations:** MBPP+'s 399-problem set is a sanitized subset of the original ~974-problem MBPP (filtering out ambiguous or malformed problems), so it is not directly comparable in raw count to the full original MBPP. ## What MBPP+ Measures EvalPlus's test-augmentation pipeline — combining LLM-based and mutation-based test-input generation — was applied to both HumanEval and MBPP. For MBPP, this involved first sanitizing the original ~974-problem dataset down to 399 well-specified problems, then generating far more test cases per problem than MBPP's original hand-written tests provided. Across the 26 LLMs evaluated in the EvalPlus paper (including GPT-4 and ChatGPT), the stricter MBPP+ test suites reduced measured pass@k scores relative to the original MBPP and, as with HumanEval+, changed which models ranked best. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Coding / rigorous entry-level Python synthesis | | Metric | pass@k on the sanitized, expanded test suite | | Number of tasks | 399 problems (sanitized subset of MBPP) | | Saturation | High for older/frontier models on original MBPP, but meaningfully lower and more discriminating on MBPP+ | | Created by | Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, Lingming Zhang (University of Illinois Urbana-Champaign) | | Source paper | [Liu et al. 2023](https://arxiv.org/abs/2305.01210) | | GitHub | [evalplus/evalplus](https://github.com/evalplus/evalplus) | ## How MBPP+ Is Scored Generated code is executed against the sanitized problem's expanded test suite; pass@k reports the probability that at least one of k independently-sampled generations passes every test case, following the same evaluation protocol as the original MBPP but with substantially more test coverage per problem. ## State-of-the-Art Results > The EvalPlus paper found that, as with HumanEval+, MBPP+'s stricter test suites reduce measured pass@k and can change model rankings relative to the original, more permissive MBPP. See the [EvalPlus leaderboard](https://github.com/evalplus/evalplus) for current per-model results. ## MBPP+ on Benchgen > No Benchgen results yet — [be the first to run MBPP+](/benchmarks/jiawei-liu-chunqiu-steven-xia-yuyao-wang-lingming-zhang-university-of-illinois-urbana-champaign/mbpp). ## MBPP+ vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **MBPP+** | Rigorously-tested entry-level Python problems | 399 | high | | [MBPP](/benchmarks/austin-et-al-google/mbpp) | Original, sparser-tested entry-level Python problems | 974 | high | | [HumanEval+](/benchmarks/jiawei-liu-chunqiu-steven-xia-yuyao-wang-lingming-zhang-university-of-illinois-urbana-champaign/humaneval) | Rigorously-tested Python function synthesis | 164 | high | MBPP+ complements HumanEval+ by focusing on simpler, more entry-level problems drawn from crowd-sourced MBPP submissions rather than HumanEval's hand-authored interview-style problems, giving a second, independently-sourced rigorous-testing signal. ## Run MBPP+ on Your Model Benchgen lets teams evaluate their own model against MBPP+'s expanded, sanitized test suite, surfacing correctness gaps that the original MBPP's sparser tests would otherwise miss. ## Frequently Asked Questions

What is MBPP+?

MBPP+ is EvalPlus's sanitized, rigorously re-tested version of MBPP, narrowing the dataset to 399 well-formed entry-level Python problems and adding far more test cases per problem.

What does a good MBPP+ score look like?

Because MBPP+'s stricter tests reduce measured pass@k relative to original MBPP, a strong MBPP+ score is one that holds up close to a model's original MBPP score rather than collapsing under stricter testing.

Who created MBPP+?

MBPP+ was created by Jiawei Liu and collaborators at the University of Illinois Urbana-Champaign as part of the EvalPlus framework, published in May 2023.

Rank	Model	Score
1	o1-preview	80.2
2	o1-mini	78.8
3	qwen2-5-coder-32b-instruct	77
4	gemini-1-5-pro	74.6
5	claude-3-5-sonnet	74.3
6	deepseek-v2-5	74.1
7	claude-3-opus	73.3
8	deepseek-v3	73
9	gpt-4o-mini	72.2
10	gpt-4o	72.2
11	gpt-3-5-turbo	69.7
12	claude-3-sonnet	69.3
13	claude-3-haiku	68.8
14	gemini-1-5-flash	67.5
15	command-r-plus	63.5
16	gemini-pro	61.4
17	llama-3-1-8b-instruct	55.6

MBPP+ — Results

MBPP+ — Evaluations