Multi-SWE-Bench

Multi-SWE-Bench

Quick answer: Multi-SWE-bench is a ByteDance-created issue-resolving benchmark that extends the SWE-bench paradigm beyond Python to seven additional languages — Java, TypeScript, JavaScript, Go, Rust, C, and C++ — with 1,632 expert-annotated GitHub instances, addressing the near-exclusive Python focus of prior issue-resolving benchmarks.

At a Glance

What it tests: Whether an LLM or coding agent can resolve real GitHub issues by generating a correct patch, evaluated across a genuinely multilingual set of codebases rather than the Python-only scope of the original SWE-bench.

Why it matters: Existing issue-resolving benchmarks focus almost exclusively on Python, leaving a blind spot for how well models generalize across programming ecosystems; Multi-SWE-bench is one of the largest efforts to close that gap with expert-verified, non-Python instances.

Known limitations: Even with 68 expert annotators curating 1,632 instances from 2,456 candidates, per-language instance counts are uneven, and multilingual issue-resolving remains meaningfully harder than Python-only SWE-bench for every model tested.

What Multi-SWE-Bench Measures

Multi-SWE-bench mirrors the SWE-bench task format — given a real GitHub issue and a snapshot of the repository, a model or agent must produce a patch that resolves the issue and passes the associated hidden tests — but extends coverage to seven languages beyond Python: Java, TypeScript, JavaScript, Go, Rust, C, and C++. The 1,632 included instances were carefully annotated from 2,456 candidates by 68 expert annotators, prioritizing accuracy and reliability of the ground-truth evaluation over raw dataset size.

The authors evaluate a range of state-of-the-art models using three representative resolution methods — Agentless, SWE-agent, and OpenHands — and report a comprehensive per-language, per-method analysis. Alongside the benchmark itself, the project launched the Multi-SWE-RL open-source community, releasing a first batch of 4,723 well-structured instances across the same seven languages specifically intended as reinforcement-learning training data for issue-resolving agents, with an open data-production pipeline for the community to expand further.

Benchmark Specifications

How Multi-SWE-Bench Is Scored

Resolution is binary per instance — a submitted patch either passes the held-out test suite (resolved) or it doesn't — and the headline metric is the resolved rate (%) aggregated per language and per resolution method (Agentless, SWE-agent, OpenHands). Because method choice materially affects results, scores are typically reported as a matrix of model × method × language rather than a single number.

State-of-the-Art Results

Resolved rates vary substantially by language and resolution method (Agentless, SWE-agent, OpenHands). See the official leaderboard for the current per-model, per-language breakdown.

Multi-SWE-Bench on Benchgen

No Benchgen results yet — be the first to run Multi-SWE-Bench.

Multi-SWE-Bench vs Other Benchmarks

Multi-SWE-bench and SWE-bench Multilingual both extend the issue-resolving paradigm beyond Python but take different tradeoffs: Multi-SWE-bench prioritizes larger instance count (1,632) with expert human-difficulty annotation, while SWE-bench Multilingual prioritizes a smaller, fast-to-run curated set (300) across more languages (9 vs. 7).

Run Multi-SWE-Bench on Your Model

Benchgen lets teams evaluate their own coding agent's multilingual issue-resolving ability across all seven languages, surfacing exactly which programming ecosystems a model's software-engineering skill fails to generalize to.

Frequently Asked Questions