# SWE-bench Multilingual ![Category](https://img.shields.io/badge/Category-Coding-1f6feb) ![Metric](https://img.shields.io/badge/Metric-Resolved%20Rate-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-300-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-low-d29922) ![Created](https://img.shields.io/badge/Created-2025-555) [![GitHub](https://img.shields.io/badge/GitHub-swe--bench%2FSWE--bench-181717?logo=github&logoColor=white)](https://github.com/swe-bench/SWE-bench) [![Dataset](https://img.shields.io/badge/Dataset-HuggingFace-ff9d00?logo=huggingface&logoColor=white)](https://huggingface.co/datasets/SWE-bench/SWE-bench_Multilingual) [![Leaderboard](https://img.shields.io/badge/Leaderboard-swebench.com-0969da)](https://www.swebench.com/multilingual.html) > **Quick answer:** SWE-bench Multilingual is the official SWE-bench team's multilingual extension of SWE-bench, consisting of 300 curated GitHub issue-resolution tasks from 42 repositories across 9 programming languages (C, C++, Go, Java, JavaScript, TypeScript, PHP, Ruby, and Rust), designed to test whether coding agents generalize beyond the Python-heavy tooling most agent frameworks are built around. ## At a Glance **What it tests:** Whether a coding agent can resolve a real GitHub issue outside Python — where many agent frameworks' Python-specific tooling (AST parsing, bug localization heuristics) can no longer be assumed to transfer. **Why it matters:** SWE-bench Verified's 500 Python issues are the industry-standard coding-agent benchmark, but Claude 3.7 Sonnet's drop from 63% (Verified) to 43% (Multilingual) under the same agent framework shows real capability gaps outside Python that a Python-only benchmark can't surface. **Known limitations:** Only 300 tasks (vs. Multi-SWE-bench's 1,632) and, at initial release, only one model (Claude 3.7 Sonnet) had been evaluated by the maintainers due to budget constraints — broader third-party evaluation fills this gap over time. ## What SWE-bench Multilingual Measures SWE-bench Multilingual follows the exact same collection strategy, dataset format, and evaluation protocol as the original SWE-bench: each task pairs a real GitHub issue with a repository snapshot at the pre-fix commit, and an agent must generate a patch that passes fail-to-pass (F2P) tests (confirming the issue is actually fixed) while preserving pass-to-pass (P2P) tests (confirming nothing else broke). The 300 tasks span 42 repositories chosen from the most-starred projects across the 9 target languages, selected via the annual Stack Overflow Developer Survey's popular-language list and GitHub-Ranking's top repositories. Unlike the larger, independently-built Multi-SWE-bench (1,632 instances across 7 languages), SWE-bench Multilingual deliberately trades scale for speed and full compatibility with existing SWE-bench infrastructure — researchers can drop it into an existing SWE-bench evaluation pipeline with no changes. At initial release, SWE-agent + Claude 3.7 Sonnet resolved 43% of tasks (with a $2.50 per-task cost limit), notably below Claude 3.7 Sonnet's 63% on SWE-bench Verified, with resolution rate varying substantially by language — Rust scored highest, C/C++ lowest. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Coding / multilingual GitHub issue resolution | | Metric | Resolved rate (%) — fail-to-pass + pass-to-pass test verification | | Number of tasks | 300 curated tasks across 42 repositories, 9 languages | | Saturation | Low — 43% resolved rate for the only model evaluated at release, vs. 63% on SWE-bench Verified | | Created by | Kabir Khandpur, Kilian Lieret, Carlos E. Jimenez, Ofir Press, John Yang (SWE-bench Team) | | GitHub | [swe-bench/SWE-bench](https://github.com/swe-bench/SWE-bench) | | Dataset | [SWE-bench/SWE-bench_Multilingual](https://huggingface.co/datasets/SWE-bench/SWE-bench_Multilingual) | | Official results | [swebench.com/multilingual.html](https://www.swebench.com/multilingual.html) | ## How SWE-bench Multilingual Is Scored A task is resolved only if the submitted patch makes all fail-to-pass tests pass (the reported bug is actually fixed) while keeping all pass-to-pass tests passing (no regressions introduced). The headline metric is the overall resolved rate (%), with the official write-up additionally breaking results down by language and by repository to show where resolution rate is strongest and weakest. ## State-of-the-Art Results | Rank | Model | Score (resolved rate, all 9 languages) | Source | Date | |---|---|---|---|---| | 1 | [Claude 3.7 Sonnet](/models/anthropic/claude-3-7-sonnet) | 43% | [SWE-bench Team](https://www.swebench.com/multilingual.html) | 2025 | *Achieved using the SWE-agent framework with a $2.50 per-task cost limit — the only model officially evaluated by the maintainers at release due to budget constraints. Resolution rate ranged from 28.57% (C/C++) to 58.14% (Rust) across the 9 languages; see the official page for the full per-language and per-repository breakdown.* ## SWE-bench Multilingual on Benchgen Benchgen has synced the official SWE-bench team's reported Claude 3.7 Sonnet baseline — see the live results table below. [Run SWE-bench Multilingual against your own model](/benchmarks/kabir-khandpur-kilian-lieret-carlos-e-jimenez-ofir-press-john-yang-swe-bench-team/swe-bench-multilingual) to add a verified entry. ## SWE-bench Multilingual vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **SWE-bench Multilingual** | Multilingual GitHub issue resolution (9 languages, official) | 300 | low | | [Multi-SWE-Bench](/benchmarks/daoguang-zan-zhirong-huang-wei-liu-et-al-bytedance/multi-swe-bench) | Multilingual GitHub issue resolution (7 languages, ByteDance) | 1,632 | low | | [SWE-bench Verified](/benchmarks/swe-bench-verified) | Python-only GitHub issue resolution | 500 | medium | SWE-bench Multilingual is the official SWE-bench team's own multilingual extension, prioritizing full pipeline compatibility and fast iteration over Multi-SWE-bench's larger, independently-curated instance count. ## Run SWE-bench Multilingual on Your Model Benchgen lets teams evaluate their own coding agent against SWE-bench Multilingual's 9-language task set, directly comparing resolution rate against the same agent's SWE-bench Verified score to quantify how much capability is Python-specific. ## Frequently Asked Questions

What is SWE-bench Multilingual?

SWE-bench Multilingual is the official SWE-bench team's benchmark of 300 curated GitHub issue-resolution tasks spanning 9 programming languages beyond Python.

What does a good SWE-bench Multilingual score look like?

The only model evaluated at release (Claude 3.7 Sonnet via SWE-agent) resolved 43% of tasks, well below its 63% on SWE-bench Verified, so scores in that range or higher represent strong, though still far from saturated, multilingual coding-agent performance.

Who created SWE-bench Multilingual?

SWE-bench Multilingual was created by Kabir Khandpur in collaboration with the official SWE-bench team (Kilian Lieret, Carlos E. Jimenez, Ofir Press, John Yang), released in 2025.

SWE-bench Multilingual — Results

SWE-bench Multilingual — Evaluations