RepoBench

RepoBench

Quick answer: RepoBench is a UC San Diego benchmark (ICLR 2024) for evaluating repository-level code auto-completion — as opposed to single-file completion — across three linked tasks: cross-file snippet retrieval, next-line code completion, and a full end-to-end pipeline, in both Python and Java.

At a Glance

What it tests: Whether a coding model or system can retrieve and correctly use context from other files in a repository to complete code accurately, rather than relying solely on the current file's contents.

Why it matters: Real-world code completion tools work inside multi-file repositories, not isolated snippets — RepoBench specifically isolates the cross-file retrieval and reasoning step that single-file benchmarks like HumanEval or MBPP cannot measure.

Known limitations: The v1.1 dataset is time-sliced from real GitHub repositories, so contamination risk grows for any model trained on code up through the dataset's collection window; results are also sensitive to context-window length allotted to cross-file snippets.

What RepoBench Measures

RepoBench decomposes repository-level completion into three interconnected tasks. RepoBench-R (Retrieval) measures whether a system can retrieve the most relevant code snippets from other files as cross-file context. RepoBench-C (Code Completion) measures next-line code prediction given both in-file and cross-file context. RepoBench-P (Pipeline) combines both — testing complex, realistic completion tasks that require retrieving relevant cross-file snippets and predicting the correct next line from them.

Within Code Completion and Pipeline tasks, RepoBench further defines three settings: cross_file_first (masks the line where a cross-file module is used for the first time), cross_file_random (masks a random cross-file usage line), and in_file (masks a line with no cross-file dependency at all, functioning as a same-file control). Comparing performance across these three settings isolates how much a model specifically benefits from — or struggles with — cross-file context.

Benchmark Specifications

How RepoBench Is Scored

Three complementary metrics are reported per setting: Exact Match (EM), the strictest measure (the predicted line must match the gold line character-for-character); Edit Similarity (ES), a softer partial-credit metric based on edit distance; and CodeBLEU (CB), which additionally accounts for code structure and data-flow, not just surface text. Scores are typically reported per context-window size (e.g., 2k/4k/8k/12k/16k tokens of cross-file context) and per setting (cross_file_first/cross_file_random/in_file), since performance varies substantially across both axes.

State-of-the-Art Results

Scores vary substantially by context-window size, setting, and language (Python vs. Java). See the official GitHub repository for the paper's full per-model, per-setting breakdown.

RepoBench on Benchgen

No Benchgen results yet — be the first to run RepoBench.

RepoBench vs Other Benchmarks

RepoBench sits between single-file benchmarks like HumanEval (no repository context at all) and full issue-resolution benchmarks like SWE-Bench Verified (requires understanding an entire codebase to fix a bug) — isolating specifically the cross-file retrieval and completion step in between.

Run RepoBench on Your Model

Benchgen lets teams evaluate their own coding model or agent's repository-level completion accuracy across context sizes and cross-file settings, surfacing exactly where cross-file reasoning breaks down rather than relying on a single-file proxy score.

Frequently Asked Questions