HumanEval+ — Results
| Rank | Model | Score |
|---|---|---|
| 1 | o1-mini | 89 |
| 2 | o1-preview | 89 |
| 3 | gpt-4o | 87.2 |
| 4 | qwen2-5-coder-32b-instruct | 87.2 |
| 5 | deepseek-v3 | 86.6 |
| 6 | gpt-4-turbo | 86.6 |
| 7 | deepseek-v2-5 | 83.5 |
| 8 | gpt-4o-mini | 83.5 |
| 9 | claude-3-5-sonnet | 81.7 |
| 10 | gemini-1-5-pro | 79.3 |
| 11 | claude-3-opus | 77.4 |
| 12 | gemini-1-5-flash | 75.6 |
| 13 | gpt-3-5-turbo | 70.7 |
| 14 | claude-3-haiku | 68.9 |
| 15 | claude-3-sonnet | 64 |
| 16 | llama-3-1-8b-instruct | 62.8 |
| 17 | command-r-plus | 56.7 |
| 18 | gemini-pro | 55.5 |
HumanEval+
1 phaseActive
EvalPlus's rigorously-tested extension of HumanEval, using 80x more test cases per problem to expose LLM-generated code that passes weak tests but is actually wrong.
Overview
HumanEval+
Quick answer: HumanEval+ is the EvalPlus framework's rigorously-augmented version of OpenAI's original HumanEval benchmark, extending the same 164 hand-written programming problems with roughly 80x more automatically-generated test cases per problem, so that code which merely passes HumanEval's sparse original tests can no longer hide subtle correctness bugs.
At a Glance
What it tests: Whether LLM-generated Python solutions to HumanEval's 164 programming problems are genuinely correct across a much wider range of inputs, not just the handful of test cases originally shipped with HumanEval.
Why it matters: The original HumanEval's test suites are sparse enough that plausible-looking but subtly incorrect code frequently passes; HumanEval+'s 80x larger test suites catch these failures, in some cases dropping measured pass@k by 19.3–28.9 percentage points and re-ordering model leaderboards.
Known limitations: HumanEval+ still inherits HumanEval's narrow scope of short, single-function Python problems, so strong HumanEval+ performance doesn't guarantee competence on larger, multi-file, real-world software engineering tasks.
What HumanEval+ Measures
EvalPlus augments existing code-generation benchmarks with large numbers of new test cases, generated through a combination of LLM-based and mutation-based test-input generation strategies. Applied to HumanEval, this produces HumanEval+: the same 164 problems, but with roughly 80x as many test cases per problem as the original release.
Across 26 evaluated LLMs (including GPT-4 and ChatGPT), the authors found that HumanEval+'s stricter testing reduces measured pass@k scores by up to 19.3–28.9 percentage points relative to original HumanEval, and can even change model rankings — the paper reports that WizardCoder-CodeLlama and Phind-CodeLlama outperform ChatGPT on HumanEval+ despite neither doing so on the original HumanEval.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Coding / rigorous Python function synthesis |
| Metric | pass@k on the augmented (80x larger) test suite |
| Number of tasks | 164 problems (same problem set as HumanEval) |
| Saturation | High for older/frontier models on original HumanEval, but meaningfully lower and more discriminating on HumanEval+ |
| Created by | Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, Lingming Zhang (University of Illinois Urbana-Champaign) |
| Source paper | Liu et al. 2023 |
| GitHub | evalplus/evalplus |
How HumanEval+ Is Scored
Generated code is executed against the expanded test suite; pass@k reports the probability that at least one of k independently-sampled generations passes every test case for a given problem, matching the original HumanEval evaluation protocol but with a much stricter bar for "passing."
State-of-the-Art Results
The original EvalPlus paper found that test insufficiency in HumanEval led to model mis-ranking: WizardCoder-CodeLlama and Phind-CodeLlama outperformed ChatGPT on HumanEval+, despite neither doing so on the original, more permissive HumanEval. See the EvalPlus leaderboard for current per-model results.
HumanEval+ on Benchgen
No Benchgen results yet — be the first to run HumanEval+.
HumanEval+ vs Other Benchmarks
| Benchmark | What it tests | Tasks | Saturation |
|---|---|---|---|
| HumanEval+ | Rigorously-tested Python function synthesis | 164 | high |
| HumanEval | Original, sparser-tested Python function synthesis | 164 | high |
| MBPP+ | Rigorously-tested entry-level Python problems | 399 | high |
| OJBench | Elite competition-level code reasoning | 232 | low |
HumanEval+ is distinguished from OJBench and other elite competitive-programming benchmarks by testing correctness rigor on the same style of introductory-to-intermediate problems as HumanEval, rather than raising the underlying problem difficulty.
Run HumanEval+ on Your Model
Benchgen lets teams evaluate their own model against HumanEval+'s rigorously-expanded test suite, surfacing correctness gaps that the original HumanEval's sparser tests would otherwise miss.