# RepoBench ![Category](https://img.shields.io/badge/Category-Coding-1f6feb) ![Metric](https://img.shields.io/badge/Metric-EM%20%2F%20ES%20%2F%20CodeBLEU-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-Python%20%2B%20Java-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-medium-d29922) ![Created](https://img.shields.io/badge/Created-Jun%202023-555) [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/abs/2306.03091) [![GitHub](https://img.shields.io/badge/GitHub-Leolty%2Frepobench-181717?logo=github&logoColor=white)](https://github.com/Leolty/repobench) [![Dataset](https://img.shields.io/badge/Dataset-HuggingFace-ff9d00?logo=huggingface&logoColor=white)](https://huggingface.co/datasets/tianyang/repobench_python_v1.1) > **Quick answer:** RepoBench is a UC San Diego benchmark (ICLR 2024) for evaluating repository-level code auto-completion — as opposed to single-file completion — across three linked tasks: cross-file snippet retrieval, next-line code completion, and a full end-to-end pipeline, in both Python and Java. ## At a Glance **What it tests:** Whether a coding model or system can retrieve and correctly use context from *other files* in a repository to complete code accurately, rather than relying solely on the current file's contents. **Why it matters:** Real-world code completion tools work inside multi-file repositories, not isolated snippets — RepoBench specifically isolates the cross-file retrieval and reasoning step that single-file benchmarks like HumanEval or MBPP cannot measure. **Known limitations:** The v1.1 dataset is time-sliced from real GitHub repositories, so contamination risk grows for any model trained on code up through the dataset's collection window; results are also sensitive to context-window length allotted to cross-file snippets. ## What RepoBench Measures RepoBench decomposes repository-level completion into three interconnected tasks. **RepoBench-R (Retrieval)** measures whether a system can retrieve the most relevant code snippets from other files as cross-file context. **RepoBench-C (Code Completion)** measures next-line code prediction given both in-file and cross-file context. **RepoBench-P (Pipeline)** combines both — testing complex, realistic completion tasks that require retrieving relevant cross-file snippets *and* predicting the correct next line from them. Within Code Completion and Pipeline tasks, RepoBench further defines three settings: `cross_file_first` (masks the line where a cross-file module is used for the first time), `cross_file_random` (masks a random cross-file usage line), and `in_file` (masks a line with no cross-file dependency at all, functioning as a same-file control). Comparing performance across these three settings isolates how much a model specifically benefits from — or struggles with — cross-file context. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Coding / repository-level code completion | | Metric | Exact Match (EM), Edit Similarity (ES), CodeBLEU (CB) | | Languages | Python and Java | | Saturation | Medium — cross-file settings remain meaningfully harder than in-file | | Created by | Tianyang Liu, Canwen Xu, Julian McAuley (UC San Diego) | | Source paper | [Liu et al. 2024](https://arxiv.org/abs/2306.03091) (ICLR 2024) | | GitHub | [Leolty/repobench](https://github.com/Leolty/repobench) | | Dataset | [tianyang/repobench_python_v1.1](https://huggingface.co/datasets/tianyang/repobench_python_v1.1) | ## How RepoBench Is Scored Three complementary metrics are reported per setting: **Exact Match (EM)**, the strictest measure (the predicted line must match the gold line character-for-character); **Edit Similarity (ES)**, a softer partial-credit metric based on edit distance; and **CodeBLEU (CB)**, which additionally accounts for code structure and data-flow, not just surface text. Scores are typically reported per context-window size (e.g., 2k/4k/8k/12k/16k tokens of cross-file context) and per setting (`cross_file_first`/`cross_file_random`/`in_file`), since performance varies substantially across both axes. ## State-of-the-Art Results > Scores vary substantially by context-window size, setting, and language (Python vs. Java). See the [official GitHub repository](https://github.com/Leolty/repobench) for the paper's full per-model, per-setting breakdown. ## RepoBench on Benchgen > No Benchgen results yet — [be the first to run RepoBench](/benchmarks/tianyang-liu-canwen-xu-julian-mcauley-uc-san-diego/repobench). ## RepoBench vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **RepoBench** | Repository-level (cross-file) code completion | Python + Java | medium | | [HumanEval](/benchmarks/humaneval) | Single-function code generation | 164 | high | | [SWE-Bench Verified](/benchmarks/swe-bench-verified) | Real GitHub issue-to-PR resolution | — | medium | RepoBench sits between single-file benchmarks like HumanEval (no repository context at all) and full issue-resolution benchmarks like SWE-Bench Verified (requires understanding an entire codebase to fix a bug) — isolating specifically the cross-file *retrieval and completion* step in between. ## Run RepoBench on Your Model Benchgen lets teams evaluate their own coding model or agent's repository-level completion accuracy across context sizes and cross-file settings, surfacing exactly where cross-file reasoning breaks down rather than relying on a single-file proxy score. ## Frequently Asked Questions

What is RepoBench?

RepoBench is a benchmark for repository-level code auto-completion, testing cross-file snippet retrieval, next-line completion, and a combined pipeline, in Python and Java.

What does a good RepoBench score look like?

Because RepoBench reports EM/ES/CodeBLEU separately per setting and context size, a strong result means maintaining performance in the harder `cross_file_first`/`cross_file_random` settings close to the `in_file` control, not just a high in-file score.

Who created RepoBench?

RepoBench was created by Tianyang Liu, Canwen Xu, and Julian McAuley at UC San Diego, published at ICLR 2024.

RepoBench — Phases

RepoBench