# NL2Repo-Bench ![Category](https://img.shields.io/badge/Category-Coding-1f6feb) ![Metric](https://img.shields.io/badge/Metric-%25%20Test%20Pass%20Rate-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-104-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-Low-d29922) ![Created](https://img.shields.io/badge/Created-Dec%202025-555) [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/abs/2512.12730) [![GitHub](https://img.shields.io/badge/GitHub-NL2RepoBench-181717?logo=github&logoColor=white)](https://github.com/multimodal-art-projection/NL2RepoBench) > **Quick answer:** NL2Repo-Bench evaluates whether a coding agent can build an entire, installable Python software repository from a single natural-language requirements document and an empty workspace — no scaffolding, no file structure, no starter code. Agents are scored by the average test pass rate their generated repository achieves against a held-out test suite. Even the strongest models average below 40%. ## At a Glance **What it tests:** Long-horizon, zero-scaffold repository construction — architecture design, dependency management, multi-module implementation, and producing a fully installable Python library from one spec document. **Why it matters:** Most coding benchmarks give agents an existing repo or a narrow function stub to fill in. NL2Repo-Bench isolates the much harder "0-to-1" problem — sustaining coherent planning and cross-file consistency over hundreds of interaction steps with no human decomposition of the task. **Known limitations:** 104 tasks, Python-only. Judged primarily via automated test suites (rule-based + LLM-based checks to block gaming via unauthorized pip/curl calls), which can still be gamed by narrow test-passing behavior that doesn't reflect real code quality. ## What NL2Repo-Bench Measures Given only a natural-language requirements document and an empty workspace, an agent must independently design the software architecture, manage its own dependencies, implement multi-module logic, and produce a fully installable Python library — end to end, with no human in the loop to decompose the problem or supervise intermediate steps. The benchmark's authors found that long-horizon repository generation remains largely unsolved even for frontier agents: the strongest models average below 40% test pass rate and rarely complete an entire repository correctly. Common failure modes include premature termination, loss of global coherence across files, fragile cross-file dependencies, and breakdowns in planning once the interaction horizon stretches past a few hundred steps. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Coding / Long-horizon agent | | Metric | Average % test pass rate against held-out test suite | | Number of tasks | 104 | | Output | Complete, installable Python library per task | | Saturation | Low (strongest agents average <40%) | | Created by | Jingzhe Ding, Shengda Long, Changxin Pu, Huan Zhou et al. | | Paper | [arXiv:2512.12730](https://arxiv.org/abs/2512.12730) | ## Benchmarks on Benchgen Scores are sourced from Zhipu AI's GLM-5.3 launch blog (August 2026), which evaluated four models under a consistent protocol (temperature=1.0, top_p=1.0, max_new_tokens=64K, 1M context, with rule-based and LLM-based anti-gaming checks). | Model | Score | Source | Date | |---|---|---|---| | [Kimi K3](/models/moonshot-ai/kimi-k3) | 58.0% | Zhipu AI GLM-5.3 launch blog | 2026-08 | | [GLM-5.3](/models/zhipuai/glm-5-3) | 58.0% | Zhipu AI GLM-5.3 launch blog | 2026-08 | | [Qwen3.8 Max](/models/alibaba/qwen3-8-max) | 55.9% | Zhipu AI GLM-5.3 launch blog | 2026-08 | | [GLM-5.2](/models/zhipu-ai/glm-5-2) | 48.9% | Zhipu AI GLM-5.3 launch blog | 2026-08 | ## Related Models & Benchmarks - Models: [GLM-5.3](/models/zhipuai/glm-5-3) · [Kimi K3](/models/moonshot-ai/kimi-k3) · [Qwen3.8 Max](/models/alibaba/qwen3-8-max) · [GLM-5.2](/models/zhipu-ai/glm-5-2) - Benchmarks: [DeepSWE](/benchmarks/datacurve/deepswe) · [Toolathlon-Verified](/benchmarks/hkust-nlp/toolathlon-verified) · [TerminalBench 2.1](/benchmarks/tbench/terminalbench-21) ## Frequently Asked Questions

What is NL2Repo-Bench?

A benchmark that tests whether a coding agent can build a complete, installable Python software repository from a single natural-language requirements document and an empty workspace, measuring the average test pass rate of the result.

What does a good score look like?

Even the strongest models score below 40% in the original paper's evaluation, and the best scores reported on Benchgen so far are in the high-50s — the benchmark is far from saturated.

Who created NL2Repo-Bench?

Jingzhe Ding, Shengda Long, Changxin Pu and co-authors, published on arXiv in December 2025 with code on GitHub under multimodal-art-projection/NL2RepoBench.

Rank	Model	Score
1	deepseek-v4-1-flash-1017	68.2
2	deepseek-v4-1-flash	64
3	deepseek-v4-1-flash	64
4	hy4-preview	58.9
5	glm-5-3	58
6	kimi-k3	58
7	deepseek-v4-flash-vision-exp	57.7
8	qwen3-8-max	55.9
9	glm-5-2	48.9
10	qwen3-8-flash-next	48.1

NL2Repo-Bench — Results

NL2Repo-Bench — Evaluations