PaperBench

PaperBench

Quick answer: PaperBench is OpenAI's benchmark for evaluating whether AI agents can replicate modern machine learning research end-to-end — reading a paper, building the codebase from scratch, and running experiments to reproduce its empirical claims — across 20 ICML 2024 papers and 8,316 gradable rubric tasks. The best publicly reported result is an IterativeAgent using o1-high with a 36-hour budget, scoring 26.0%.

At a Glance

What it tests: Whether an agent can go from a research paper to a working, correctly-behaving reproduction of its experiments — not just answer questions about the paper. Why it matters: This is one of the most direct public tests of long-horizon AI research engineering: the output is a functioning codebase and results, graded against a detailed rubric, not a short text answer. Known limitations: Scores are reported as agent-scaffold + underlying-model combinations, not model-only results — the "harness" (agent design) matters as much as the base model.

What PaperBench Measures

PaperBench gives an agent a modern ML paper and asks it to replicate the paper's experiments from scratch: understanding the method, writing the code, running training/evaluation, and reproducing the reported results. Each paper is broken into a detailed rubric — 8,316 individually gradable tasks across the 20 papers — covering both code-level correctness and whether final results match the paper's claims.

This makes PaperBench a test of long-horizon, open-ended engineering rather than short-task code generation: agents must plan a multi-day project, debug their own implementation, and interpret ambiguous specifications the way a human researcher would.

Benchmark Specifications

How PaperBench Is Scored

Each paper's reproduction attempt is graded against its detailed rubric by an automated grader, producing a per-paper replication score; the benchmark score is the average across all 20 papers, reported as a percentage (0–100%). Because the rubric checks both implementation details and final results, partial credit is common — very few runs reproduce a paper fully.

State-of-the-Art Results

Scores are reported as agent scaffold + underlying model combinations — the "harness" (IterativeAgent vs. BasicAgent) has a large effect independent of the base model.

The paper reports these scores remain well below the human baseline for the same tasks.

PaperBench on Benchgen

No Benchgen results yet — be the first to run PaperBench.

PaperBench vs Other Benchmarks

Run PaperBench on Your Model

Because PaperBench scores depend heavily on the agent scaffold, not just the base model, Benchgen lets you track harness + model combinations over time with version-controlled, regression-tracked results — useful for isolating whether a gain came from a better model or a better scaffold.

Related Models & Benchmarks

• DeepSeek-R1
• FrontierMath
• SWE-bench Verified

Frequently Asked Questions