SWE-bench Multilingual

SWE-bench Multilingual

Quick answer: SWE-bench Multilingual is the official SWE-bench team's multilingual extension of SWE-bench, consisting of 300 curated GitHub issue-resolution tasks from 42 repositories across 9 programming languages (C, C++, Go, Java, JavaScript, TypeScript, PHP, Ruby, and Rust), designed to test whether coding agents generalize beyond the Python-heavy tooling most agent frameworks are built around.

At a Glance

What it tests: Whether a coding agent can resolve a real GitHub issue outside Python — where many agent frameworks' Python-specific tooling (AST parsing, bug localization heuristics) can no longer be assumed to transfer.

Why it matters: SWE-bench Verified's 500 Python issues are the industry-standard coding-agent benchmark, but Claude 3.7 Sonnet's drop from 63% (Verified) to 43% (Multilingual) under the same agent framework shows real capability gaps outside Python that a Python-only benchmark can't surface.

Known limitations: Only 300 tasks (vs. Multi-SWE-bench's 1,632) and, at initial release, only one model (Claude 3.7 Sonnet) had been evaluated by the maintainers due to budget constraints — broader third-party evaluation fills this gap over time.

What SWE-bench Multilingual Measures

SWE-bench Multilingual follows the exact same collection strategy, dataset format, and evaluation protocol as the original SWE-bench: each task pairs a real GitHub issue with a repository snapshot at the pre-fix commit, and an agent must generate a patch that passes fail-to-pass (F2P) tests (confirming the issue is actually fixed) while preserving pass-to-pass (P2P) tests (confirming nothing else broke). The 300 tasks span 42 repositories chosen from the most-starred projects across the 9 target languages, selected via the annual Stack Overflow Developer Survey's popular-language list and GitHub-Ranking's top repositories.

Unlike the larger, independently-built Multi-SWE-bench (1,632 instances across 7 languages), SWE-bench Multilingual deliberately trades scale for speed and full compatibility with existing SWE-bench infrastructure — researchers can drop it into an existing SWE-bench evaluation pipeline with no changes. At initial release, SWE-agent + Claude 3.7 Sonnet resolved 43% of tasks (with a $2.50 per-task cost limit), notably below Claude 3.7 Sonnet's 63% on SWE-bench Verified, with resolution rate varying substantially by language — Rust scored highest, C/C++ lowest.

Benchmark Specifications

How SWE-bench Multilingual Is Scored

A task is resolved only if the submitted patch makes all fail-to-pass tests pass (the reported bug is actually fixed) while keeping all pass-to-pass tests passing (no regressions introduced). The headline metric is the overall resolved rate (%), with the official write-up additionally breaking results down by language and by repository to show where resolution rate is strongest and weakest.

State-of-the-Art Results

Achieved using the SWE-agent framework with a $2.50 per-task cost limit — the only model officially evaluated by the maintainers at release due to budget constraints. Resolution rate ranged from 28.57% (C/C++) to 58.14% (Rust) across the 9 languages; see the official page for the full per-language and per-repository breakdown.

SWE-bench Multilingual on Benchgen

Benchgen has synced the official SWE-bench team's reported Claude 3.7 Sonnet baseline — see the live results table below. Run SWE-bench Multilingual against your own model to add a verified entry.

SWE-bench Multilingual vs Other Benchmarks

SWE-bench Multilingual is the official SWE-bench team's own multilingual extension, prioritizing full pipeline compatibility and fast iteration over Multi-SWE-bench's larger, independently-curated instance count.

Run SWE-bench Multilingual on Your Model

Benchgen lets teams evaluate their own coding agent against SWE-bench Multilingual's 9-language task set, directly comparing resolution rate against the same agent's SWE-bench Verified score to quantify how much capability is Python-specific.

Frequently Asked Questions