# PaperBench ![Category](https://img.shields.io/badge/Category-Agent-1f6feb) ![Metric](https://img.shields.io/badge/Metric-Replication%20Score-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-8%2C316-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-Low-red) ![Created](https://img.shields.io/badge/Created-Apr%202025-555) [![Paper](https://img.shields.io/badge/Paper-arXiv%202504.01848-b31b1b)](https://arxiv.org/abs/2504.01848) [![GitHub](https://img.shields.io/badge/GitHub-openai%2Ffrontier--evals-181717?logo=github&logoColor=white)](https://github.com/openai/frontier-evals/tree/main/project/paperbench) > **Quick answer:** PaperBench is OpenAI's benchmark for evaluating whether AI agents can replicate modern machine learning research end-to-end — reading a paper, building the codebase from scratch, and running experiments to reproduce its empirical claims — across 20 ICML 2024 papers and 8,316 gradable rubric tasks. The best publicly reported result is an IterativeAgent using o1-high with a 36-hour budget, scoring 26.0%. --- ## At a Glance **What it tests:** Whether an agent can go from a research paper to a working, correctly-behaving reproduction of its experiments — not just answer questions about the paper. **Why it matters:** This is one of the most direct public tests of long-horizon AI research engineering: the output is a functioning codebase and results, graded against a detailed rubric, not a short text answer. **Known limitations:** Scores are reported as agent-scaffold + underlying-model combinations, not model-only results — the "harness" (agent design) matters as much as the base model. ## What PaperBench Measures PaperBench gives an agent a modern ML paper and asks it to replicate the paper's experiments from scratch: understanding the method, writing the code, running training/evaluation, and reproducing the reported results. Each paper is broken into a detailed rubric — 8,316 individually gradable tasks across the 20 papers — covering both code-level correctness and whether final results match the paper's claims. This makes PaperBench a test of long-horizon, open-ended engineering rather than short-task code generation: agents must plan a multi-day project, debug their own implementation, and interpret ambiguous specifications the way a human researcher would. ## Benchmark Specifications | Field | Value | |---|---| | Papers | 20 (ICML 2024) | | Gradable rubric tasks | 8,316 | | Task category | Agentic AI research replication | | Metric | Average replication score (0–100%) | | Released | April 2025 | | Created by | Giulio Starace, Oliver Jaffe, Dane Sherburn, and collaborators (OpenAI) | ## How PaperBench Is Scored Each paper's reproduction attempt is graded against its detailed rubric by an automated grader, producing a per-paper replication score; the benchmark score is the average across all 20 papers, reported as a percentage (0–100%). Because the rubric checks both implementation details and final results, partial credit is common — very few runs reproduce a paper fully. ## State-of-the-Art Results Scores are reported as **agent scaffold + underlying model** combinations — the "harness" (IterativeAgent vs. BasicAgent) has a large effect independent of the base model. | Rank | Agent / Harness | Model | Score | Source | |---|---|---|---:|---| | 1 | IterativeAgent (36h budget) | o1-high | 26.0% ± 0.3 | [PaperBench README](https://github.com/openai/frontier-evals/blob/main/project/paperbench/README.md) | | 2 | IterativeAgent (24h budget) | o1-high | 24.4% ± 0.7 | [PaperBench README](https://github.com/openai/frontier-evals/blob/main/project/paperbench/README.md) | | 3 | BasicAgent | claude-3.5-sonnet | 21.0% ± 0.8 | [PaperBench README](https://github.com/openai/frontier-evals/blob/main/project/paperbench/README.md) | | 4 | IterativeAgent | claude-3.5-sonnet | 16.1% ± 0.1 | [PaperBench README](https://github.com/openai/frontier-evals/blob/main/project/paperbench/README.md) | | 5 | BasicAgent | o1-high | 13.2% ± 0.3 | [PaperBench README](https://github.com/openai/frontier-evals/blob/main/project/paperbench/README.md) | | 6 | IterativeAgent | o3-mini-high | 8.5% ± 0.8 | [PaperBench README](https://github.com/openai/frontier-evals/blob/main/project/paperbench/README.md) | | 7 | BasicAgent | deepseek-r1 | 6.0% ± 0.3 | [PaperBench README](https://github.com/openai/frontier-evals/blob/main/project/paperbench/README.md) | | 8 | BasicAgent | gpt-4o | 4.1% ± 0.1 | [PaperBench README](https://github.com/openai/frontier-evals/blob/main/project/paperbench/README.md) | | 9 | BasicAgent | gemini-2.0-flash | 3.2% ± 0.2 | [PaperBench README](https://github.com/openai/frontier-evals/blob/main/project/paperbench/README.md) | | 10 | BasicAgent | o3-mini-high | 2.6% ± 0.2 | [PaperBench README](https://github.com/openai/frontier-evals/blob/main/project/paperbench/README.md) | *The paper reports these scores remain well below the human baseline for the same tasks.* ## PaperBench on Benchgen > No Benchgen results yet — [be the first to run PaperBench](/benchmarks/paperbench). ## PaperBench vs Other Benchmarks | Benchmark | What it tests | Saturation | |---|---|---| | PaperBench | End-to-end ML research replication | Low | | [FrontierMath](/benchmarks/frontiermath) | Frontier mathematical research problems | — | | [SWE-bench Verified](/benchmarks/swe-bench-verified) | Real-world software engineering fixes | — | ## Run PaperBench on Your Model Because PaperBench scores depend heavily on the agent scaffold, not just the base model, Benchgen lets you track harness + model combinations over time with version-controlled, regression-tracked results — useful for isolating whether a gain came from a better model or a better scaffold. ## Related Models & Benchmarks - [DeepSeek-R1](/models/deepseek/deepseek-r1-0528) - [FrontierMath](/benchmarks/frontiermath) - [SWE-bench Verified](/benchmarks/swe-bench-verified) ## Frequently Asked Questions

What is PaperBench?

PaperBench is OpenAI's benchmark for evaluating whether AI agents can replicate ML research papers end-to-end, from reading the paper through running experiments that reproduce its claims.

What does a good score look like?

The best publicly reported result is 26.0% (IterativeAgent, o1-high, 36-hour budget) — scores are still well below full replication, so any result above ~20% represents a leading result as of the benchmark's release.

Who created PaperBench?

PaperBench was created by Giulio Starace, Oliver Jaffe, Dane Sherburn, and collaborators at OpenAI, published in April 2025.

Rank	Model	Score
1	o1	26
2	claude-3-5-sonnet	21
3	claude-3-5-sonnet	16.1
4	o1	13.2
5	o3-mini	8.5
6	deepseek-r1-0528	6
7	gpt-4o	4.1
8	gemini-2-0-flash	3.2
9	o3-mini	2.6

PaperBench — Results

PaperBench — Evaluations