NL2Repo-Bench — Results
| Rank | Model | Score |
|---|---|---|
| 1 | glm-5-3 | 58 |
| 2 | kimi-k3 | 58 |
| 3 | qwen3-8-max | 55.9 |
| 4 | glm-5-2 | 48.9 |
| 5 | qwen3-8-flash-next | 48.1 |
NL2Repo-Bench
1 phaseActive
Long-horizon coding benchmark — agents build a complete, installable Python library from a single NL requirements doc, scored by test pass rate.
Overview
NL2Repo-Bench
Quick answer: NL2Repo-Bench evaluates whether a coding agent can build an entire, installable Python software repository from a single natural-language requirements document and an empty workspace — no scaffolding, no file structure, no starter code. Agents are scored by the average test pass rate their generated repository achieves against a held-out test suite. Even the strongest models average below 40%.
At a Glance
What it tests: Long-horizon, zero-scaffold repository construction — architecture design, dependency management, multi-module implementation, and producing a fully installable Python library from one spec document.
Why it matters: Most coding benchmarks give agents an existing repo or a narrow function stub to fill in. NL2Repo-Bench isolates the much harder "0-to-1" problem — sustaining coherent planning and cross-file consistency over hundreds of interaction steps with no human decomposition of the task.
Known limitations: 104 tasks, Python-only. Judged primarily via automated test suites (rule-based + LLM-based checks to block gaming via unauthorized pip/curl calls), which can still be gamed by narrow test-passing behavior that doesn't reflect real code quality.
What NL2Repo-Bench Measures
Given only a natural-language requirements document and an empty workspace, an agent must independently design the software architecture, manage its own dependencies, implement multi-module logic, and produce a fully installable Python library — end to end, with no human in the loop to decompose the problem or supervise intermediate steps.
The benchmark's authors found that long-horizon repository generation remains largely unsolved even for frontier agents: the strongest models average below 40% test pass rate and rarely complete an entire repository correctly. Common failure modes include premature termination, loss of global coherence across files, fragile cross-file dependencies, and breakdowns in planning once the interaction horizon stretches past a few hundred steps.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Coding / Long-horizon agent |
| Metric | Average % test pass rate against held-out test suite |
| Number of tasks | 104 |
| Output | Complete, installable Python library per task |
| Saturation | Low (strongest agents average <40%) |
| Created by | Jingzhe Ding, Shengda Long, Changxin Pu, Huan Zhou et al. |
| Paper | arXiv:2512.12730 |
Benchmarks on Benchgen
Scores are sourced from Zhipu AI's GLM-5.3 launch blog (August 2026), which evaluated four models under a consistent protocol (temperature=1.0, top_p=1.0, max_new_tokens=64K, 1M context, with rule-based and LLM-based anti-gaming checks).
| Model | Score | Source | Date |
|---|---|---|---|
| Kimi K3 | 58.0% | Zhipu AI GLM-5.3 launch blog | 2026-08 |
| GLM-5.3 | 58.0% | Zhipu AI GLM-5.3 launch blog | 2026-08 |
| Qwen3.8 Max | 55.9% | Zhipu AI GLM-5.3 launch blog | 2026-08 |
| GLM-5.2 | 48.9% | Zhipu AI GLM-5.3 launch blog | 2026-08 |
Related Models & Benchmarks
- Models: GLM-5.3 · Kimi K3 · Qwen3.8 Max · GLM-5.2
- Benchmarks: DeepSWE · Toolathlon-Verified · TerminalBench 2.1