# HumanEval+ ![Category](https://img.shields.io/badge/Category-Coding-1f6feb) ![Metric](https://img.shields.io/badge/Metric-Pass%40k-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-164-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-high-d29922) ![Created](https://img.shields.io/badge/Created-May%202023-555) [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/abs/2305.01210) [![GitHub](https://img.shields.io/badge/GitHub-evalplus%2Fevalplus-181717?logo=github&logoColor=white)](https://github.com/evalplus/evalplus) > **Quick answer:** HumanEval+ is the EvalPlus framework's rigorously-augmented version of OpenAI's original HumanEval benchmark, extending the same 164 hand-written programming problems with roughly 80x more automatically-generated test cases per problem, so that code which merely passes HumanEval's sparse original tests can no longer hide subtle correctness bugs. ## At a Glance **What it tests:** Whether LLM-generated Python solutions to HumanEval's 164 programming problems are genuinely correct across a much wider range of inputs, not just the handful of test cases originally shipped with HumanEval. **Why it matters:** The original HumanEval's test suites are sparse enough that plausible-looking but subtly incorrect code frequently passes; HumanEval+'s 80x larger test suites catch these failures, in some cases dropping measured pass@k by 19.3–28.9 percentage points and re-ordering model leaderboards. **Known limitations:** HumanEval+ still inherits HumanEval's narrow scope of short, single-function Python problems, so strong HumanEval+ performance doesn't guarantee competence on larger, multi-file, real-world software engineering tasks. ## What HumanEval+ Measures EvalPlus augments existing code-generation benchmarks with large numbers of new test cases, generated through a combination of LLM-based and mutation-based test-input generation strategies. Applied to HumanEval, this produces HumanEval+: the same 164 problems, but with roughly 80x as many test cases per problem as the original release. Across 26 evaluated LLMs (including GPT-4 and ChatGPT), the authors found that HumanEval+'s stricter testing reduces measured pass@k scores by up to 19.3–28.9 percentage points relative to original HumanEval, and can even change model rankings — the paper reports that WizardCoder-CodeLlama and Phind-CodeLlama outperform ChatGPT on HumanEval+ despite neither doing so on the original HumanEval. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Coding / rigorous Python function synthesis | | Metric | pass@k on the augmented (80x larger) test suite | | Number of tasks | 164 problems (same problem set as HumanEval) | | Saturation | High for older/frontier models on original HumanEval, but meaningfully lower and more discriminating on HumanEval+ | | Created by | Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, Lingming Zhang (University of Illinois Urbana-Champaign) | | Source paper | [Liu et al. 2023](https://arxiv.org/abs/2305.01210) | | GitHub | [evalplus/evalplus](https://github.com/evalplus/evalplus) | ## How HumanEval+ Is Scored Generated code is executed against the expanded test suite; pass@k reports the probability that at least one of k independently-sampled generations passes every test case for a given problem, matching the original HumanEval evaluation protocol but with a much stricter bar for "passing." ## State-of-the-Art Results > The original EvalPlus paper found that test insufficiency in HumanEval led to model mis-ranking: WizardCoder-CodeLlama and Phind-CodeLlama outperformed ChatGPT on HumanEval+, despite neither doing so on the original, more permissive HumanEval. See the [EvalPlus leaderboard](https://github.com/evalplus/evalplus) for current per-model results. ## HumanEval+ on Benchgen > No Benchgen results yet — [be the first to run HumanEval+](/benchmarks/jiawei-liu-chunqiu-steven-xia-yuyao-wang-lingming-zhang-university-of-illinois-urbana-champaign/humaneval). ## HumanEval+ vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **HumanEval+** | Rigorously-tested Python function synthesis | 164 | high | | [HumanEval](/benchmarks/openai/humaneval) | Original, sparser-tested Python function synthesis | 164 | high | | [MBPP+](/benchmarks/jiawei-liu-chunqiu-steven-xia-yuyao-wang-lingming-zhang-university-of-illinois-urbana-champaign/mbpp) | Rigorously-tested entry-level Python problems | 399 | high | | [OJBench](/benchmarks/zhexu-wang-yiping-liu-yejie-wang-wenyang-he-bofei-gao-muxi-diao-yanxu-chen-kelin-fu-flood-sung-et-al-moonshot-ai/ojbench) | Elite competition-level code reasoning | 232 | low | HumanEval+ is distinguished from OJBench and other elite competitive-programming benchmarks by testing correctness rigor on the same style of introductory-to-intermediate problems as HumanEval, rather than raising the underlying problem difficulty. ## Run HumanEval+ on Your Model Benchgen lets teams evaluate their own model against HumanEval+'s rigorously-expanded test suite, surfacing correctness gaps that the original HumanEval's sparser tests would otherwise miss. ## Frequently Asked Questions

What is HumanEval+?

HumanEval+ is EvalPlus's rigorously-augmented version of HumanEval, testing the same 164 Python problems against roughly 80x more test cases per problem to catch subtly incorrect generated code.

What does a good HumanEval+ score look like?

Because HumanEval+'s stricter tests can drop pass@k by up to 19.3–28.9 percentage points versus original HumanEval, a strong HumanEval+ score is one that holds up close to a model's original HumanEval score rather than collapsing under stricter testing.

Who created HumanEval+?

HumanEval+ was created by Jiawei Liu and collaborators at the University of Illinois Urbana-Champaign as part of the EvalPlus framework, published in May 2023.

Rank	Model	Score
1	o1-mini	89
2	o1-preview	89
3	gpt-4o	87.2
4	qwen2-5-coder-32b-instruct	87.2
5	deepseek-v3	86.6
6	gpt-4-turbo	86.6
7	deepseek-v2-5	83.5
8	gpt-4o-mini	83.5
9	claude-3-5-sonnet	81.7
10	gemini-1-5-pro	79.3
11	claude-3-opus	77.4
12	gemini-1-5-flash	75.6
13	gpt-3-5-turbo	70.7
14	claude-3-haiku	68.9
15	claude-3-sonnet	64
16	llama-3-1-8b-instruct	62.8
17	command-r-plus	56.7
18	gemini-pro	55.5

HumanEval+ — Results

HumanEval+ — Evaluations