HumanEval

HumanEval

Quick answer: HumanEval (Chen et al., July 2021) is OpenAI's seminal Python code generation benchmark consisting of 164 hand-crafted programming problems. Models generate function implementations from docstrings and are scored by executing against hidden unit tests, making it the most widely-cited and historically significant LLM coding evaluation. The benchmark is now heavily saturated — most frontier models exceed 90% pass@1.

At a Glance

What it tests: Python function synthesis from natural language docstrings, covering algorithms, string manipulation, data structures, and basic math.

Why it matters: HumanEval established the standard methodology for evaluating LLM code generation — generate code, execute against tests, measure pass@1 — and remains the baseline reference point for comparing models across generations.

Known limitations: The 164-problem set is heavily saturated for frontier models (most score 90%+), limiting discriminability at the top end. Contamination risk is high given the dataset is public and widely cited in training papers.

What HumanEval Measures

HumanEval presents each model with a Python function signature and a natural language docstring describing the expected behavior. The model must generate the complete function body. Correctness is determined by executing the model's output against a set of hidden unit tests — if all tests pass, the sample is considered correct.

The primary metric is pass@1 — the probability that a single sample passes all tests with greedy decoding. Higher-order metrics (pass@10, pass@100) measure whether a model can solve the problem given multiple attempts, which is useful for understanding the model's exploration capability but less commonly reported for comparison.

Problems range from simple list operations and string processing to more complex algorithmic tasks. The benchmark intentionally avoids problems requiring external libraries, keeping all solutions implementable with Python's standard library.

Benchmark Specifications

How HumanEval Is Scored

For each problem, the model generates a function body given the signature and docstring. The generated code is sandboxed and executed against a private set of unit tests. Pass@1 is the fraction of problems where the first (greedy) sample passes all unit tests. Due to the high saturation of modern models — nearly all frontier models score above 88% — HumanEval is increasingly used as a sanity check rather than a primary discriminator.

FAQ

What is HumanEval? HumanEval is a 164-problem Python coding benchmark introduced by OpenAI in 2021. It evaluates a model's ability to generate correct function implementations from docstrings, scored by executing the generated code against unit tests.

Is HumanEval still useful for comparing LLMs? For frontier models, HumanEval is heavily saturated — most score above 88% — making it a poor discriminator between top models. More challenging successors like BigCodeBench or LiveCodeBench are now preferred for fine-grained comparisons.

What is a good HumanEval score? Scores above 90% are now common for frontier models. The original Codex (2021) scored 28.8%, while GPT-4 was the first to widely exceed 80%. Today even small instruction-tuned models routinely exceed 70%.