ProgramBench — Results
| Rank | Model | Score |
|---|---|---|
| 1 | kimi-k3 | 77.8 |
| 2 | kimi-k3-1-pro | 31.5 |
| 3 | deepseek-v4-1-flash-1017 | 29.1 |
| 4 | deepseek-v4-1-flash | 20.3 |
| 5 | claude-opus-5-5 | 18.5 |
| 6 | claude-opus-5-5 | 18.5 |
| 7 | hy4-preview | 17.5 |
| 8 | claude-fable-5-1 | 7 |
| 9 | claude-fable-5-1 | 7 |
| 10 | claude-sonnet-5-5 | 6.5 |
| 11 | claude-sonnet-5-5 | 6.5 |
| 12 | gpt-6-astra | 5.5 |
| 13 | gpt-6-astra | 5.5 |
| 14 | gpt-6-1-sol | 3 |
| 15 | gpt-6-1-sol | 3 |
ProgramBench
1 phaseActive
Academic coding-agent benchmark by Yang et al. (Meta Superintelligence Labs, Stanford, Harvard): rebuild 200 command-line programs from a binary and its documentation, graded by hidden behavioral tests. Metric: % fully resolved.
Overview
ProgramBench
Quick answer: ProgramBench asks whether a coding agent can rebuild a command-line program from scratch, given only its compiled binary and its documentation. It is an academic benchmark by Yang et al. (Meta Superintelligence Labs, Stanford, Harvard); Vals AI runs it independently and lists it as academic. The best fully resolved rate on Vals AI is 18.5% (Claude Opus 5.5, October 2026).
At a Glance
What it tests: Writing a whole program that behaves like an existing one, from an execute-only binary and its docs. Decompilers, disassemblers and internet lookups are not allowed.
Why it matters: It is a long-horizon software task with no reference code to copy: the agent has to infer the behavior, design the program and get the edge cases right.
Known limitations: The headline metric is strict (a task counts only if every hidden test passes), so scores are low and close together. Model developers often report a different metric or their own harness, so self-reported numbers are usually not comparable with the fully resolved rate.
What ProgramBench Measures
Each of the 200 tasks gives the agent a compiled command-line program and its documentation. The agent writes an implementation from scratch, which is then run against hidden behavioral tests (over 248,000 across the benchmark) that compare it with the original program.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Coding (agentic) |
| Tasks | 200 public tasks, over 248,000 hidden behavioral tests |
| Metric | % fully resolved (all hidden tests pass) |
| Secondary metrics | Almost resolved (at least 95% of tests pass), raw pass rate |
| Saturation | Low |
| Created by | Yang et al. (Meta Superintelligence Labs, Stanford University, Harvard University) |
| Paper | ProgramBench: Can Language Models Rebuild Programs From Scratch? |
| Independent leaderboard | Vals AI (listed as academic) |
How ProgramBench Is Scored
A task is fully resolved only when the submitted program passes all of its hidden behavioral tests. The headline score is the share of the 200 tasks that are fully resolved. Two secondary numbers are reported: almost resolved (at least 95% of the tests pass) and the raw pass rate (the average share of tests passed per task). The authors warn that the average pass rate on its own is misleading, because a program can pass most tests and still not work.
State-of-the-Art Results
Independent runs by Vals AI, fully resolved, updated 2026-10-12:
| Rank | Model | Fully resolved | Source | Date |
|---|---|---|---|---|
| 1 | Claude Opus 5.5 | 18.5% | Vals AI | 2026-10 |
| 2 | Claude Fable 5.1 | 7.0% | Vals AI | 2026-10 |
| 3 | Claude Sonnet 5.5 | 6.5% | Vals AI | 2026-10 |
| 4 | GPT-6 Astra | 5.5% | Vals AI | 2026-10 |
| 5 | GPT-6.1 Sol | 3.0% | Vals AI | 2026-10 |
Reported by model developers (different harness or metric, not comparable with the table above):
| Model | Score | Note | Source |
|---|---|---|---|
| DeepSeek-V4.1-Flash | 20.3% | Developer-reported | Model card |
| Hy4 Preview | 17.5% | Developer-reported | Model card |
| Kimi K3 | 77.8% | Run with the Kimi Code harness; metric not stated, not the fully resolved rate | Kimi K3 report |
ProgramBench on Benchgen
No Benchgen runs yet. The scores above are mirrored from Vals AI and model developers, not re-run by Benchgen.
ProgramBench vs Other Benchmarks
| Benchmark | What it tests | Saturation |
|---|---|---|
| ProgramBench | Rebuilding a program from its binary and docs | Low |
| BigCodeBench | Practical code generation | Medium |
| LiveCodeBench | Contamination-resistant competitive coding | Medium |
| DeepSWE | Software engineering agent tasks | Low |
Run ProgramBench on Your Model
The 200 tasks and the evaluation setup are public on GitHub, so ProgramBench can be reproduced outside the authors' setup. A run needs a sandbox where the agent can execute the binary and where the program it builds is checked against the hidden tests.