# ProgramBench ![Category](https://img.shields.io/badge/Category-Coding-1f6feb) ![Metric](https://img.shields.io/badge/Metric-%25%20Fully%20Resolved-2da44e) ![Saturation](https://img.shields.io/badge/Saturation-Low-d29922) ![Created](https://img.shields.io/badge/Created-2026-555) [![Website](https://img.shields.io/badge/Website-programbench.com-0969da)](https://programbench.com) [![arXiv](https://img.shields.io/badge/arXiv-2605.03546-b31b1b)](https://arxiv.org/abs/2605.03546) [![GitHub](https://img.shields.io/badge/GitHub-facebookresearch%2FProgramBench-555)](https://github.com/facebookresearch/ProgramBench) > **Quick answer:** ProgramBench asks whether a coding agent can rebuild a command-line program from scratch, given only its compiled binary and its documentation. It is an academic benchmark by Yang et al. (Meta Superintelligence Labs, Stanford, Harvard); Vals AI runs it independently and lists it as academic. The best fully resolved rate on Vals AI is 18.5% (Claude Opus 5.5, October 2026). ## At a Glance **What it tests:** Writing a whole program that behaves like an existing one, from an execute-only binary and its docs. Decompilers, disassemblers and internet lookups are not allowed. **Why it matters:** It is a long-horizon software task with no reference code to copy: the agent has to infer the behavior, design the program and get the edge cases right. **Known limitations:** The headline metric is strict (a task counts only if every hidden test passes), so scores are low and close together. Model developers often report a different metric or their own harness, so self-reported numbers are usually not comparable with the fully resolved rate. ## What ProgramBench Measures Each of the 200 tasks gives the agent a compiled command-line program and its documentation. The agent writes an implementation from scratch, which is then run against hidden behavioral tests (over 248,000 across the benchmark) that compare it with the original program. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Coding (agentic) | | Tasks | 200 public tasks, over 248,000 hidden behavioral tests | | Metric | % fully resolved (all hidden tests pass) | | Secondary metrics | Almost resolved (at least 95% of tests pass), raw pass rate | | Saturation | Low | | Created by | Yang et al. (Meta Superintelligence Labs, Stanford University, Harvard University) | | Paper | [ProgramBench: Can Language Models Rebuild Programs From Scratch?](https://arxiv.org/abs/2605.03546) | | Independent leaderboard | [Vals AI](https://www.vals.ai/benchmarks/programbench) (listed as academic) | ## How ProgramBench Is Scored A task is fully resolved only when the submitted program passes all of its hidden behavioral tests. The headline score is the share of the 200 tasks that are fully resolved. Two secondary numbers are reported: almost resolved (at least 95% of the tests pass) and the raw pass rate (the average share of tests passed per task). The authors warn that the average pass rate on its own is misleading, because a program can pass most tests and still not work. ## State-of-the-Art Results Independent runs by Vals AI, fully resolved, updated 2026-10-12: | Rank | Model | Fully resolved | Source | Date | |---|---|---|---|---| | 1 | [Claude Opus 5.5](/models/anthropic/claude-opus-5-5) | 18.5% | [Vals AI](https://www.vals.ai/benchmarks/programbench) | 2026-10 | | 2 | [Claude Fable 5.1](/models/anthropic/claude-fable-5-1) | 7.0% | [Vals AI](https://www.vals.ai/benchmarks/programbench) | 2026-10 | | 3 | [Claude Sonnet 5.5](/models/anthropic/claude-sonnet-5-5) | 6.5% | [Vals AI](https://www.vals.ai/benchmarks/programbench) | 2026-10 | | 4 | [GPT-6 Astra](/models/openai/gpt-6-astra) | 5.5% | [Vals AI](https://www.vals.ai/benchmarks/programbench) | 2026-10 | | 5 | [GPT-6.1 Sol](/models/openai/gpt-6-1-sol) | 3.0% | [Vals AI](https://www.vals.ai/benchmarks/programbench) | 2026-10 | Reported by model developers (different harness or metric, not comparable with the table above): | Model | Score | Note | Source | |---|---|---|---| | [DeepSeek-V4.1-Flash](/models/deepseek/deepseek-v4-1-flash) | 20.3% | Developer-reported | [Model card](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash) | | [Hy4 Preview](/models/hungyuan/hy4-preview) | 17.5% | Developer-reported | [Model card](https://huggingface.co/tencent/Hy4-preview) | | [Kimi K3](/models/moonshot-ai/kimi-k3) | 77.8% | Run with the Kimi Code harness; metric not stated, not the fully resolved rate | [Kimi K3 report](https://www.kimi.ai/blog/kimi-k3) | ## ProgramBench on Benchgen > No Benchgen runs yet. The scores above are mirrored from Vals AI and model developers, not re-run by Benchgen. ## ProgramBench vs Other Benchmarks | Benchmark | What it tests | Saturation | |---|---|---| | **ProgramBench** | Rebuilding a program from its binary and docs | Low | | [BigCodeBench](/benchmarks/bigcode/bigcodebench) | Practical code generation | Medium | | [LiveCodeBench](/benchmarks/jain-et-al/livecodebench) | Contamination-resistant competitive coding | Medium | | [DeepSWE](/benchmarks/datacurve/deepswe) | Software engineering agent tasks | Low | ## Run ProgramBench on Your Model The 200 tasks and the evaluation setup are public on [GitHub](https://github.com/facebookresearch/ProgramBench), so ProgramBench can be reproduced outside the authors' setup. A run needs a sandbox where the agent can execute the binary and where the program it builds is checked against the hidden tests. ## Frequently Asked Questions

What is ProgramBench?

ProgramBench is an academic benchmark that asks a coding agent to rebuild a command-line program from scratch, using only the compiled binary and its documentation. It has 200 tasks graded by hidden behavioral tests.

What does a good score look like on ProgramBench?

On the strict fully resolved metric, 18.5% (Claude Opus 5.5 on Vals AI, October 2026) is the best independent result, and most frontier models resolve under 10% of the tasks. Higher numbers in model announcements usually use a different metric, such as the average test pass rate.

Who created ProgramBench?

ProgramBench was created by John Yang, Kilian Lieret, Jeffrey Ma, Parth Thakkar, Dmitrii Pedchenko, Sten Sootla, Emily McMilin, Pengcheng Yin, Rui Hou, Gabriel Synnaeve, Diyi Yang and Ofir Press (Meta Superintelligence Labs, Stanford University, Harvard University). Vals AI did not create it: it runs the benchmark independently and lists it as academic.

Rank	Model	Score
1	kimi-k3	77.8
2	kimi-k3-1-pro	31.5
3	deepseek-v4-1-flash-1017	29.1
4	deepseek-v4-1-flash	20.3
5	claude-opus-5-5	18.5
6	claude-opus-5-5	18.5
7	hy4-preview	17.5
8	claude-fable-5-1	7
9	claude-fable-5-1	7
10	claude-sonnet-5-5	6.5
11	claude-sonnet-5-5	6.5
12	gpt-6-astra	5.5
13	gpt-6-astra	5.5
14	gpt-6-1-sol	3
15	gpt-6-1-sol	3

ProgramBench — Results

ProgramBench — Leaderboard

ProgramBench Leaderboard