# Multi-SWE-Bench ![Category](https://img.shields.io/badge/Category-Coding-1f6feb) ![Metric](https://img.shields.io/badge/Metric-Resolved%20Rate-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-1%2C632-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-low-d29922) ![Created](https://img.shields.io/badge/Created-Apr%202025-555) [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/abs/2504.02605) [![GitHub](https://img.shields.io/badge/GitHub-multi--swe--bench-181717?logo=github&logoColor=white)](https://github.com/multi-swe-bench/multi-swe-bench) > **Quick answer:** Multi-SWE-bench is a ByteDance-created issue-resolving benchmark that extends the SWE-bench paradigm beyond Python to seven additional languages — Java, TypeScript, JavaScript, Go, Rust, C, and C++ — with 1,632 expert-annotated GitHub instances, addressing the near-exclusive Python focus of prior issue-resolving benchmarks. ## At a Glance **What it tests:** Whether an LLM or coding agent can resolve real GitHub issues by generating a correct patch, evaluated across a genuinely multilingual set of codebases rather than the Python-only scope of the original SWE-bench. **Why it matters:** Existing issue-resolving benchmarks focus almost exclusively on Python, leaving a blind spot for how well models generalize across programming ecosystems; Multi-SWE-bench is one of the largest efforts to close that gap with expert-verified, non-Python instances. **Known limitations:** Even with 68 expert annotators curating 1,632 instances from 2,456 candidates, per-language instance counts are uneven, and multilingual issue-resolving remains meaningfully harder than Python-only SWE-bench for every model tested. ## What Multi-SWE-Bench Measures Multi-SWE-bench mirrors the SWE-bench task format — given a real GitHub issue and a snapshot of the repository, a model or agent must produce a patch that resolves the issue and passes the associated hidden tests — but extends coverage to seven languages beyond Python: Java, TypeScript, JavaScript, Go, Rust, C, and C++. The 1,632 included instances were carefully annotated from 2,456 candidates by 68 expert annotators, prioritizing accuracy and reliability of the ground-truth evaluation over raw dataset size. The authors evaluate a range of state-of-the-art models using three representative resolution methods — Agentless, SWE-agent, and OpenHands — and report a comprehensive per-language, per-method analysis. Alongside the benchmark itself, the project launched the **Multi-SWE-RL** open-source community, releasing a first batch of 4,723 well-structured instances across the same seven languages specifically intended as reinforcement-learning training data for issue-resolving agents, with an open data-production pipeline for the community to expand further. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Coding / multilingual GitHub issue resolution | | Metric | Resolved rate (%) — patch passes hidden test suite | | Number of tasks | 1,632 instances across 7 languages (Java, TypeScript, JavaScript, Go, Rust, C, C++) | | Saturation | Low — multilingual resolution rates trail Python-only SWE-bench for all tested models | | Created by | Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Linhao Zhang, et al. (ByteDance) | | Source paper | [Zan et al. 2025](https://arxiv.org/abs/2504.02605) | | GitHub | [multi-swe-bench/multi-swe-bench](https://github.com/multi-swe-bench/multi-swe-bench) | ## How Multi-SWE-Bench Is Scored Resolution is binary per instance — a submitted patch either passes the held-out test suite (resolved) or it doesn't — and the headline metric is the resolved rate (%) aggregated per language and per resolution method (Agentless, SWE-agent, OpenHands). Because method choice materially affects results, scores are typically reported as a matrix of model × method × language rather than a single number. ## State-of-the-Art Results > Resolved rates vary substantially by language and resolution method (Agentless, SWE-agent, OpenHands). See the [official leaderboard](https://multi-swe-bench.github.io/) for the current per-model, per-language breakdown. ## Multi-SWE-Bench on Benchgen > No Benchgen results yet — [be the first to run Multi-SWE-Bench](/benchmarks/daoguang-zan-zhirong-huang-wei-liu-et-al-bytedance/multi-swe-bench). ## Multi-SWE-Bench vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **Multi-SWE-Bench** | Multilingual GitHub issue resolution (7 languages) | 1,632 | low | | [SWE-bench Multilingual](/benchmarks/kabir-khandpur-kilian-lieret-carlos-e-jimenez-ofir-press-john-yang-swe-bench-team/swe-bench-multilingual) | Multilingual GitHub issue resolution (9 languages) | 300 | low | | [SWE-bench Verified](/benchmarks/swe-bench-verified) | Python-only GitHub issue resolution | 500 | medium | Multi-SWE-bench and SWE-bench Multilingual both extend the issue-resolving paradigm beyond Python but take different tradeoffs: Multi-SWE-bench prioritizes larger instance count (1,632) with expert human-difficulty annotation, while SWE-bench Multilingual prioritizes a smaller, fast-to-run curated set (300) across more languages (9 vs. 7). ## Run Multi-SWE-Bench on Your Model Benchgen lets teams evaluate their own coding agent's multilingual issue-resolving ability across all seven languages, surfacing exactly which programming ecosystems a model's software-engineering skill fails to generalize to. ## Frequently Asked Questions

What is Multi-SWE-Bench?

Multi-SWE-bench is a ByteDance benchmark that extends SWE-bench-style GitHub issue resolution to seven non-Python languages, using 1,632 expert-annotated instances.

What does a good Multi-SWE-Bench score look like?

Resolved rates on Multi-SWE-bench trail the equivalent Python-only SWE-bench Verified scores for every tested model, so maintaining a high resolved rate consistently across all seven languages represents strong performance.

Who created Multi-SWE-Bench?

Multi-SWE-bench was created by Daoguang Zan and collaborators at ByteDance, published in April 2025.

Rank	Model	Score
1	claude-sonnet-3-7	9.25
2	claude-3-5-sonnet	7.97
3	o1	6.31
4	o3-mini-high	5.27
5	deepseek-v3	4.72
6	gpt-4o	3.8
7	qwen2-5-72b-instruct	2.51

Multi-SWE-Bench — Results

Multi-SWE-Bench — Phases

Multi-SWE-Bench