SWE-bench Verified

SWE-bench Verified

Quick answer: SWE-bench Verified is a human-validated subset of 500 real GitHub issues drawn from 12 popular Python repositories, created by Jimenez et al. (ICLR 2024) in collaboration with OpenAI. A model scores a point for each issue where its generated code patch passes all existing unit tests. The current SOTA is 95.0% (Claude Fable 5), up from 1.96% when the benchmark launched in 2023. Benchgen tracks 50+ models on this leaderboard.

At a Glance

What it tests: Whether a coding agent can read a real GitHub issue and produce a code patch that makes the repository's existing test suite pass — end-to-end, inside a sandboxed execution environment.

Why it matters: SWE-bench Verified is the de facto industry standard for measuring code agent capability on real-world software engineering tasks. Every major lab reports scores here; it separates models that can assist with isolated code snippets from those that can autonomously resolve multi-file bugs.

Known limitations: Scores are highly sensitive to the scaffolding (agent loop, tool access, retry budget) used around the model — not just the model itself. A model's vendor-reported score may use a heavily optimised harness, making cross-model comparison tricky without controlled evaluation conditions.

What SWE-bench Verified Measures

SWE-bench Verified presents a model with two inputs: the full source code of a real Python repository at the state it was in when a bug was filed, and the text of the corresponding GitHub issue. The model must produce a unified diff patch that, when applied to the codebase, causes all related unit tests to pass.

The "Verified" qualifier distinguishes this subset from the original SWE-bench full set (2,294 tasks). Human annotators — working in collaboration with OpenAI — reviewed each of the 500 selected instances to confirm that the problem description is unambiguous, the test patches correctly capture the required fix, and the task is solvable given only the information visible to the model. This filtering removes noisy tasks where success or failure depended on unclear specs rather than model capability.

Resolving a SWE-bench Verified instance typically requires understanding how changes in one file propagate through others, reading long context windows spanning entire modules, and coordinating multi-step edits across classes and functions. This is qualitatively harder than single-function code generation benchmarks like HumanEval.

Benchmark Specifications

How SWE-bench Verified Is Scored

A model's score is the percentage of the 500 tasks where its submitted patch, applied to the repository, causes every relevant test in the existing test suite to pass. No partial credit: an instance is either resolved (1) or not (0). The final number is reported as a percentage (e.g. 81.42%) or as a decimal (0.8142) depending on the source.

Results vary substantially based on the evaluation harness. The official leaderboard at swebench.com accepts results from any scaffold — including multi-agent pipelines, RAG systems, and multi-rollout review approaches — which explains why some scores are far higher than what a model achieves with a minimal setup. The swebench.com "Bash Only" section specifically compares models under a controlled minimal scaffold (a single ReAct loop with bash access) to enable apples-to-apples model comparisons.

A score above 50% is considered strong by 2026 standards. Scores below 30% generally reflect models without specialised coding agent training.

SWE-bench Verified Score Progression (2023–2026)

Progress on this benchmark has been among the fastest of any major AI evaluation:

The benchmark has compressed nearly 3 years of expected progress into roughly 30 months. This rate of improvement means SWE-bench Verified may approach saturation (95%+) within one to two more model generations, after which harder variants like SWE-bench Pro will become the primary differentiator.

SWE-bench Verified on Benchgen

No Benchgen results yet — be the first to run SWE-bench Verified.

SWE-bench Verified vs Other Coding Benchmarks

SWE-bench Verified is the hardest of the common coding benchmarks in terms of real-world fidelity. HumanEval measures narrow function synthesis and is saturated above 90% by frontier models; SWE-bench Verified requires understanding entire codebases, navigating long contexts, and coordinating multi-file edits — tasks much closer to what a developer actually does.

Run SWE-bench Verified on Your Model

Vendor-reported SWE-bench Verified scores tell you what a model can do in a vendor-optimised setup with unlimited retries and a custom scaffold. Benchgen lets your team run SWE-bench Verified against your specific model version — with your agent framework, your tool access, and your prompt — so you can track whether a model update actually improves real coding task resolution rather than just a leaderboard number. Run repeatedly across releases to catch regressions before they reach production.

Related Models & Benchmarks

• Top performers: Claude Opus 4.6 · Claude Sonnet 4.6 · GPT-5 · Gemini 3 Pro · Grok 4
• Related benchmarks: HumanEval · LiveCodeBench · GPQA Diamond

Frequently Asked Questions

Benchmark definition paraphrased from Jimenez et al. 2024. Leaderboard scores sourced from published model cards, technical reports, and llm-stats.com, attributed inline. Last updated 2026-07-14.