MM-Mind2Web
1 phaseActive
Screenshot-paired version of Mind2Web spanning 2,022 tasks and 14,193 actions across seen/unseen websites and domains, for evaluating multimodal web agents.
Overview
MM-Mind2Web
Quick answer: Multimodal-Mind2Web (MM-Mind2Web) pairs every HTML page in the original Mind2Web web-agent benchmark with its corresponding webpage screenshot, enabling large multimodal models like GPT-4V to be evaluated as generalist web agents that see and act on real websites rather than reasoning over HTML alone.
At a Glance
What it tests: Whether an LMM-powered web agent can plan and execute multi-step actions (click, type, select) across real websites, evaluated on seen websites, unseen websites within seen domains, and entirely unseen domains.
Why it matters: MM-Mind2Web was introduced alongside SeeAct to demonstrate that GPT-4V has strong potential as a generalist web agent when its textual plans can be grounded into concrete on-page actions — but that grounding, not planning, remains the key bottleneck.
Known limitations: Offline evaluation uses cached HTML/screenshot snapshots, so agent actions aren't validated against a live, changing website; the paper's online evaluation setting (via live websites) partially addresses this but is harder to reproduce at scale.
What MM-Mind2Web Measures
MM-Mind2Web aligns every HTML document in the original Mind2Web dataset with its corresponding webpage screenshot image, removing the need to download Mind2Web's ~300GB raw dump just to get visual context. It preserves Mind2Web's three evaluation splits — test_task (seen websites), test_website (unseen websites within seen domains), and test_domain (entirely unseen domains) — spanning a combined 2,022 tasks and 14,193 individual actions.
The accompanying SeeAct system uses large multimodal models (initially GPT-4V) to interpret each webpage screenshot, reason about the next action in natural language, and then ground that plan into a concrete UI action using either HTML-structure-based grounding, visual grounding, or a hybrid of both. The paper found GPT-4V could complete 51.1% of live-website tasks when its plans were manually grounded, substantially outperforming text-only LLMs and earlier fine-tuned web-agent baselines, but grounding strategies like set-of-mark prompting proved less effective than expected, leaving a real gap versus oracle grounding.
MM-Mind2Web has become a standard reference for evaluating multimodal web agents' ability to generalize across websites and domains they haven't seen during training.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Agent (multimodal web navigation) |
| Metric | Element accuracy, step success rate, and task success rate across three generalization splits |
| Number of tasks | 2,022 tasks (14,193 actions) across test_task, test_website, and test_domain splits |
| Saturation | Low — grounding remains a major bottleneck even for strong LMMs |
| Created by | Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, Yu Su |
| Source paper | Zheng et al. 2024 (ICML 2024) |
| Code | GitHub |
| Dataset | Hugging Face |
How MM-Mind2Web Is Scored
Offline evaluation compares predicted actions (element + operation) against ground-truth action sequences, reporting element accuracy and step-level success rate. The paper's online evaluation setting additionally runs agents on live websites and measures end-to-end task success rate, either with manual grounding verification or fully automated grounding.
State-of-the-Art Results
Scores vary significantly by grounding strategy (HTML-based, visual, hybrid) and whether evaluation is offline or on live websites. See the official repository for current standings.
MM-Mind2Web on Benchgen
No Benchgen results yet — be the first to run MM-Mind2Web.
MM-Mind2Web vs Other Benchmarks
| Benchmark | What it tests | Tasks | Saturation |
|---|---|---|---|
| MM-Mind2Web | Multimodal web-agent grounding across seen/unseen sites & domains | 2,022 | low |
| MMSearch | Multimodal AI search-engine capability | 300 | low |
| Design2Code | Screenshot-to-code front-end generation | 484 | low |
MM-Mind2Web focuses on grounded action-taking on real websites, complementing MMSearch's focus on search-specific reasoning and Design2Code's focus on generating front-end code from visual designs.
Run MM-Mind2Web on Your Model
Benchgen lets teams run their own multimodal web agent stack against MM-Mind2Web's seen/unseen splits on demand, tracking element accuracy and task success rate over time to catch generalization regressions as agent architectures change.
Related Models & Benchmarks
Explore related agent and multimodal benchmarks like MMSearch and Design2Code to compare web-agent grounding with search reasoning and code-generation evaluation.