# MM-Mind2Web ![Category](https://img.shields.io/badge/Category-Agent-1f6feb) ![Metric](https://img.shields.io/badge/Metric-Task%20Success%20Rate-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-2%2C022-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-low-d29922) ![Created](https://img.shields.io/badge/Created-Jan%202024-555) [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/abs/2401.01614) [![GitHub](https://img.shields.io/badge/GitHub-Code-181717?logo=github&logoColor=white)](https://github.com/OSU-NLP-Group/SeeAct) > **Quick answer:** Multimodal-Mind2Web (MM-Mind2Web) pairs every HTML page in the original Mind2Web web-agent benchmark with its corresponding webpage screenshot, enabling large multimodal models like GPT-4V to be evaluated as generalist web agents that see and act on real websites rather than reasoning over HTML alone. ## At a Glance **What it tests:** Whether an LMM-powered web agent can plan and execute multi-step actions (click, type, select) across real websites, evaluated on seen websites, unseen websites within seen domains, and entirely unseen domains. **Why it matters:** MM-Mind2Web was introduced alongside SeeAct to demonstrate that GPT-4V has strong potential as a generalist web agent when its textual plans can be grounded into concrete on-page actions — but that grounding, not planning, remains the key bottleneck. **Known limitations:** Offline evaluation uses cached HTML/screenshot snapshots, so agent actions aren't validated against a live, changing website; the paper's online evaluation setting (via live websites) partially addresses this but is harder to reproduce at scale. ## What MM-Mind2Web Measures MM-Mind2Web aligns every HTML document in the original Mind2Web dataset with its corresponding webpage screenshot image, removing the need to download Mind2Web's ~300GB raw dump just to get visual context. It preserves Mind2Web's three evaluation splits — test_task (seen websites), test_website (unseen websites within seen domains), and test_domain (entirely unseen domains) — spanning a combined 2,022 tasks and 14,193 individual actions. The accompanying SeeAct system uses large multimodal models (initially GPT-4V) to interpret each webpage screenshot, reason about the next action in natural language, and then ground that plan into a concrete UI action using either HTML-structure-based grounding, visual grounding, or a hybrid of both. The paper found GPT-4V could complete 51.1% of live-website tasks when its plans were manually grounded, substantially outperforming text-only LLMs and earlier fine-tuned web-agent baselines, but grounding strategies like set-of-mark prompting proved less effective than expected, leaving a real gap versus oracle grounding. MM-Mind2Web has become a standard reference for evaluating multimodal web agents' ability to generalize across websites and domains they haven't seen during training. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Agent (multimodal web navigation) | | Metric | Element accuracy, step success rate, and task success rate across three generalization splits | | Number of tasks | 2,022 tasks (14,193 actions) across test_task, test_website, and test_domain splits | | Saturation | Low — grounding remains a major bottleneck even for strong LMMs | | Created by | Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, Yu Su | | Source paper | [Zheng et al. 2024](https://arxiv.org/abs/2401.01614) (ICML 2024) | | Code | [GitHub](https://github.com/OSU-NLP-Group/SeeAct) | | Dataset | [Hugging Face](https://huggingface.co/datasets/osunlp/Multimodal-Mind2Web) | ## How MM-Mind2Web Is Scored Offline evaluation compares predicted actions (element + operation) against ground-truth action sequences, reporting element accuracy and step-level success rate. The paper's online evaluation setting additionally runs agents on live websites and measures end-to-end task success rate, either with manual grounding verification or fully automated grounding. ## State-of-the-Art Results > Scores vary significantly by grounding strategy (HTML-based, visual, hybrid) and whether evaluation is offline or on live websites. See the [official repository](https://github.com/OSU-NLP-Group/SeeAct) for current standings. ## MM-Mind2Web on Benchgen > No Benchgen results yet — [be the first to run MM-Mind2Web](/benchmarks/zheng-et-al-ohio-state-university/mm-mind2web). ## MM-Mind2Web vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **MM-Mind2Web** | Multimodal web-agent grounding across seen/unseen sites & domains | 2,022 | low | | [MMSearch](/benchmarks/jiang-et-al-cuhk-mmlab/mmsearch) | Multimodal AI search-engine capability | 300 | low | | [Design2Code](/benchmarks/si-et-al-stanford-university/design2code) | Screenshot-to-code front-end generation | 484 | low | MM-Mind2Web focuses on grounded action-taking on real websites, complementing MMSearch's focus on search-specific reasoning and Design2Code's focus on generating front-end code from visual designs. ## Run MM-Mind2Web on Your Model Benchgen lets teams run their own multimodal web agent stack against MM-Mind2Web's seen/unseen splits on demand, tracking element accuracy and task success rate over time to catch generalization regressions as agent architectures change. ## Related Models & Benchmarks Explore related agent and multimodal benchmarks like [MMSearch](/benchmarks/jiang-et-al-cuhk-mmlab/mmsearch) and [Design2Code](/benchmarks/si-et-al-stanford-university/design2code) to compare web-agent grounding with search reasoning and code-generation evaluation. ## Frequently Asked Questions

What is MM-Mind2Web?

MM-Mind2Web (Multimodal-Mind2Web) pairs the Mind2Web web-agent benchmark's HTML pages with webpage screenshots, enabling large multimodal models to be evaluated as generalist web agents on 2,022 tasks across seen and unseen websites/domains.

What does a good MM-Mind2Web score look like?

In the original SeeAct paper, GPT-4V completed 51.1% of live-website tasks with manual grounding verification, substantially ahead of text-only LLM baselines — fully automated grounding strategies still trail this oracle-assisted number.

Who created MM-Mind2Web?

MM-Mind2Web was created by Boyuan Zheng and collaborators at The Ohio State University, alongside the SeeAct system, accepted at ICML 2024.

MM-Mind2Web — Phases

MM-Mind2Web