SWE-bench Pro

SWE-bench Pro

Quick answer: SWE-bench Pro is a harder-difficulty variant of the SWE-bench coding agent benchmark, designed to test model performance on more challenging real-world GitHub issue resolution tasks than SWE-bench Verified. It is evaluated using the mini-swe-agent scaffold. Fugu Ultra achieves 73.7% as of June 2026.

At a Glance

What it tests: Whether a coding agent can resolve harder, real-world GitHub issues by producing patches that pass all relevant unit tests.

Why it matters: SWE-bench Verified is approaching saturation at the top end; SWE-bench Pro provides a more discriminating signal for frontier and multi-agent coding systems.

Known limitations: Evaluated with mini-swe-agent scaffold — scores reflect both model capability and the specific scaffolding. Cross-benchmark methodology details are provided in the Sakana Fugu technical report.

What SWE-bench Pro Measures

SWE-bench Pro follows the same core protocol as SWE-bench Verified: a model receives a GitHub repository at a specific commit and the text of a real issue, and must produce a patch that causes all relevant tests to pass. The "Pro" designation selects for harder tasks, targeting the segment of GitHub issues where current frontier models still struggle — providing a more informative separation between top-tier systems.

Scores are reported as the percentage of tasks resolved, using the mini-swe-agent as the agent scaffold. This lightweight scaffolding provides a more reproducible comparison point than custom multi-agent pipelines.

Benchmark Specifications

State-of-the-Art Results

All scores above use the mini-swe-agent scaffold, as reported by Sakana AI in the Fugu technical report.