# SWE-bench Multimodal ![Category](https://img.shields.io/badge/Category-Coding-1f6feb) ![Metric](https://img.shields.io/badge/Metric-%25%20Resolved-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-617-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-low-d29922) ![Created](https://img.shields.io/badge/Created-Oct%202024-555) [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/abs/2410.03859) [![GitHub](https://img.shields.io/badge/GitHub-Code-181717?logo=github&logoColor=white)](https://github.com/swe-bench/SWE-bench) > **Quick answer:** SWE-bench Multimodal (SWE-bench M) extends the original SWE-bench beyond Python-only, text-heavy issues to 617 real-world bug-fix tasks in visual, user-facing JavaScript software — every task includes at least one image in its problem statement or unit tests, testing whether AI coding systems generalize to visual software domains. ## At a Glance **What it tests:** Whether an autonomous coding system can resolve real GitHub issues in JavaScript libraries used for web interface design, diagramming, data visualization, syntax highlighting, and interactive mapping — where the bug report often includes a screenshot, diagram, or rendered UI image. **Why it matters:** The original SWE-bench uses only Python repositories with text-only problem statements, leaving front-end, game development, and other visual/JavaScript-heavy domains entirely untested; SWE-bench M reveals that top-performing SWE-bench systems don't automatically generalize to these unrepresented domains. **Known limitations:** SWE-bench M is limited to JavaScript libraries and 17 specific repositories, so it doesn't cover the full diversity of visual software domains (e.g., native mobile UI, game engines written in other languages). ## What SWE-bench Multimodal Measures SWE-bench Multimodal features 617 task instances collected from 17 JavaScript libraries covering web interface design, diagramming, data visualization, syntax highlighting, and interactive mapping. Unlike the original SWE-bench, which draws exclusively from Python repositories with predominantly text-only problem statements, every SWE-bench M task instance contains at least one image — a screenshot of the bug, a rendered diagram, or similar visual context — that a system must correctly interpret to produce a passing fix. The paper's analysis found that top-performing systems on the original SWE-bench struggled notably on SWE-bench M, revealing real limitations in both visual problem-solving and cross-language generalization (moving from Python to JavaScript). Notably, SWE-agent's flexible, language-agnostic design allowed it to substantially outperform alternative systems, resolving 12% of task instances compared to 6% for the next-best system at the time of the paper's release — underscoring how much headroom remains in visual, cross-language software engineering. SWE-bench Multimodal is maintained as part of the broader SWE-bench family alongside SWE-bench Verified, SWE-bench Lite, and SWE-bench Multilingual, with an actively tracked public leaderboard. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Coding (visual, JavaScript software bug-fixing) | | Metric | Percentage of task instances resolved (tests pass) | | Number of tasks | 617 task instances across 17 JavaScript libraries | | Saturation | Low — best systems resolved only ~12% of tasks at release | | Created by | John Yang, Carlos E. Jimenez, Alex L. Zhang, Kilian Lieret, and collaborators | | Source paper | [Yang et al. 2024](https://arxiv.org/abs/2410.03859) (ICLR 2025) | | Code | [GitHub](https://github.com/swe-bench/SWE-bench) | | Dataset | [Hugging Face](https://huggingface.co/datasets/SWE-bench/SWE-bench_Multimodal) | ## How SWE-bench Multimodal Is Scored Each task instance is scored as resolved or unresolved based on whether the generated patch passes the repository's associated test suite (FAIL_TO_PASS and PASS_TO_PASS test criteria), following the same pass/fail evaluation methodology used across the SWE-bench family, then reported as an overall percentage resolved. ## State-of-the-Art Results > Scores vary significantly by agent scaffold (e.g., SWE-agent vs. other systems) and underlying LLM. See the [official leaderboard](https://www.swebench.com/multimodal.html) for current standings. ## SWE-bench Multimodal on Benchgen > No Benchgen results yet — [be the first to run SWE-bench Multimodal](/benchmarks/yang-et-al-princeton-university-stanford-university/swe-bench-multimodal). ## SWE-bench Multimodal vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **SWE-bench Multimodal** | Visual JavaScript bug-fixing | 617 | low | | [Design2Code](/benchmarks/si-et-al-stanford-university/design2code) | Screenshot-to-code webpage generation | 484 | medium | | [MM-Mind2Web](/benchmarks/zheng-et-al-ohio-state-university/mm-mind2web) | Multimodal web-agent action grounding | 2,022 | low | SWE-bench Multimodal tests fixing existing visual software bugs, complementing Design2Code's focus on generating new front-end code from a design and MM-Mind2Web's focus on taking actions within a live webpage. ## Run SWE-bench Multimodal on Your Model Benchgen lets teams run their own coding agent against SWE-bench Multimodal's 617 visual JavaScript tasks on demand, tracking resolution rate over time to catch regressions in cross-language, visual bug-fixing capability. ## Related Models & Benchmarks Explore related coding and agent benchmarks like [Design2Code](/benchmarks/si-et-al-stanford-university/design2code) and [MM-Mind2Web](/benchmarks/zheng-et-al-ohio-state-university/mm-mind2web) to compare visual bug-fixing with front-end code generation and web-agent action-taking. ## Frequently Asked Questions

What is SWE-bench Multimodal?

SWE-bench Multimodal is a 617-task extension of SWE-bench that tests whether AI coding systems can fix real bugs in visual, user-facing JavaScript software, with every task including at least one embedded image.

What does a good SWE-bench Multimodal score look like?

At release, the best-performing system (SWE-agent) resolved only about 12% of tasks, so any system meaningfully exceeding that resolution rate represents a strong, state-of-the-art result.

Who created SWE-bench Multimodal?

SWE-bench Multimodal was created by John Yang, Carlos E. Jimenez, and collaborators at Princeton University and Stanford University, accepted at ICLR 2025.

SWE-bench Multimodal — Leaderboard

SWE-bench Multimodal Leaderboard