MBPP+ — Results
| Rank | Model | Score |
|---|---|---|
| 1 | o1-preview | 80.2 |
| 2 | o1-mini | 78.8 |
| 3 | qwen2-5-coder-32b-instruct | 77 |
| 4 | gemini-1-5-pro | 74.6 |
| 5 | claude-3-5-sonnet | 74.3 |
| 6 | deepseek-v2-5 | 74.1 |
| 7 | claude-3-opus | 73.3 |
| 8 | deepseek-v3 | 73 |
| 9 | gpt-4o-mini | 72.2 |
| 10 | gpt-4o | 72.2 |
| 11 | gpt-3-5-turbo | 69.7 |
| 12 | claude-3-sonnet | 69.3 |
| 13 | claude-3-haiku | 68.8 |
| 14 | gemini-1-5-flash | 67.5 |
| 15 | command-r-plus | 63.5 |
| 16 | gemini-pro | 61.4 |
| 17 | llama-3-1-8b-instruct | 55.6 |
MBPP+
1 phaseActive
EvalPlus's rigorously-tested, sanitized subset of MBPP's entry-level Python problems, using many more test cases per problem than the original benchmark.
Overview
MBPP+
Quick answer: MBPP+ is the EvalPlus framework's rigorously re-tested and sanitized version of MBPP (Mostly Basic Python Problems), narrowing the original dataset to 399 well-formed entry-level problems and dramatically expanding the test cases per problem so that superficially-passing but subtly-wrong code can no longer hide.
At a Glance
What it tests: Whether LLM-generated Python solutions to entry-level programming problems are genuinely correct across a much wider range of inputs than MBPP's original, sparser test suites checked for.
Why it matters: Like HumanEval+, MBPP+ exposed that a meaningful share of previously "passing" model solutions on MBPP were subtly wrong, and that stricter testing can reorder which models actually generate the most reliable code.
Known limitations: MBPP+'s 399-problem set is a sanitized subset of the original ~974-problem MBPP (filtering out ambiguous or malformed problems), so it is not directly comparable in raw count to the full original MBPP.
What MBPP+ Measures
EvalPlus's test-augmentation pipeline — combining LLM-based and mutation-based test-input generation — was applied to both HumanEval and MBPP. For MBPP, this involved first sanitizing the original ~974-problem dataset down to 399 well-specified problems, then generating far more test cases per problem than MBPP's original hand-written tests provided.
Across the 26 LLMs evaluated in the EvalPlus paper (including GPT-4 and ChatGPT), the stricter MBPP+ test suites reduced measured pass@k scores relative to the original MBPP and, as with HumanEval+, changed which models ranked best.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Coding / rigorous entry-level Python synthesis |
| Metric | pass@k on the sanitized, expanded test suite |
| Number of tasks | 399 problems (sanitized subset of MBPP) |
| Saturation | High for older/frontier models on original MBPP, but meaningfully lower and more discriminating on MBPP+ |
| Created by | Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, Lingming Zhang (University of Illinois Urbana-Champaign) |
| Source paper | Liu et al. 2023 |
| GitHub | evalplus/evalplus |
How MBPP+ Is Scored
Generated code is executed against the sanitized problem's expanded test suite; pass@k reports the probability that at least one of k independently-sampled generations passes every test case, following the same evaluation protocol as the original MBPP but with substantially more test coverage per problem.
State-of-the-Art Results
The EvalPlus paper found that, as with HumanEval+, MBPP+'s stricter test suites reduce measured pass@k and can change model rankings relative to the original, more permissive MBPP. See the EvalPlus leaderboard for current per-model results.
MBPP+ on Benchgen
No Benchgen results yet — be the first to run MBPP+.
MBPP+ vs Other Benchmarks
| Benchmark | What it tests | Tasks | Saturation |
|---|---|---|---|
| MBPP+ | Rigorously-tested entry-level Python problems | 399 | high |
| MBPP | Original, sparser-tested entry-level Python problems | 974 | high |
| HumanEval+ | Rigorously-tested Python function synthesis | 164 | high |
MBPP+ complements HumanEval+ by focusing on simpler, more entry-level problems drawn from crowd-sourced MBPP submissions rather than HumanEval's hand-authored interview-style problems, giving a second, independently-sourced rigorous-testing signal.
Run MBPP+ on Your Model
Benchgen lets teams evaluate their own model against MBPP+'s expanded, sanitized test suite, surfacing correctness gaps that the original MBPP's sparser tests would otherwise miss.