SWE-bench Multimodal

SWE-bench Multimodal

Quick answer: SWE-bench Multimodal (SWE-bench M) extends the original SWE-bench beyond Python-only, text-heavy issues to 617 real-world bug-fix tasks in visual, user-facing JavaScript software — every task includes at least one image in its problem statement or unit tests, testing whether AI coding systems generalize to visual software domains.

At a Glance

What it tests: Whether an autonomous coding system can resolve real GitHub issues in JavaScript libraries used for web interface design, diagramming, data visualization, syntax highlighting, and interactive mapping — where the bug report often includes a screenshot, diagram, or rendered UI image.

Why it matters: The original SWE-bench uses only Python repositories with text-only problem statements, leaving front-end, game development, and other visual/JavaScript-heavy domains entirely untested; SWE-bench M reveals that top-performing SWE-bench systems don't automatically generalize to these unrepresented domains.

Known limitations: SWE-bench M is limited to JavaScript libraries and 17 specific repositories, so it doesn't cover the full diversity of visual software domains (e.g., native mobile UI, game engines written in other languages).

What SWE-bench Multimodal Measures

SWE-bench Multimodal features 617 task instances collected from 17 JavaScript libraries covering web interface design, diagramming, data visualization, syntax highlighting, and interactive mapping. Unlike the original SWE-bench, which draws exclusively from Python repositories with predominantly text-only problem statements, every SWE-bench M task instance contains at least one image — a screenshot of the bug, a rendered diagram, or similar visual context — that a system must correctly interpret to produce a passing fix.

The paper's analysis found that top-performing systems on the original SWE-bench struggled notably on SWE-bench M, revealing real limitations in both visual problem-solving and cross-language generalization (moving from Python to JavaScript). Notably, SWE-agent's flexible, language-agnostic design allowed it to substantially outperform alternative systems, resolving 12% of task instances compared to 6% for the next-best system at the time of the paper's release — underscoring how much headroom remains in visual, cross-language software engineering.

SWE-bench Multimodal is maintained as part of the broader SWE-bench family alongside SWE-bench Verified, SWE-bench Lite, and SWE-bench Multilingual, with an actively tracked public leaderboard.

Benchmark Specifications

How SWE-bench Multimodal Is Scored

Each task instance is scored as resolved or unresolved based on whether the generated patch passes the repository's associated test suite (FAIL_TO_PASS and PASS_TO_PASS test criteria), following the same pass/fail evaluation methodology used across the SWE-bench family, then reported as an overall percentage resolved.

State-of-the-Art Results

Scores vary significantly by agent scaffold (e.g., SWE-agent vs. other systems) and underlying LLM. See the official leaderboard for current standings.

SWE-bench Multimodal on Benchgen

No Benchgen results yet — be the first to run SWE-bench Multimodal.

SWE-bench Multimodal vs Other Benchmarks

SWE-bench Multimodal tests fixing existing visual software bugs, complementing Design2Code's focus on generating new front-end code from a design and MM-Mind2Web's focus on taking actions within a live webpage.

Run SWE-bench Multimodal on Your Model

Benchgen lets teams run their own coding agent against SWE-bench Multimodal's 617 visual JavaScript tasks on demand, tracking resolution rate over time to catch regressions in cross-language, visual bug-fixing capability.

Related Models & Benchmarks

Explore related coding and agent benchmarks like Design2Code and MM-Mind2Web to compare visual bug-fixing with front-end code generation and web-agent action-taking.

Frequently Asked Questions