# Mind2Web ![Category](https://img.shields.io/badge/Category-Agent-1f6feb) ![Metric](https://img.shields.io/badge/Metric-Step%20Success%20Rate-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-2%2C350-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-Low-d29922) ![Created](https://img.shields.io/badge/Created-Jun%202023-555) [![Paper](https://img.shields.io/badge/Paper-arXiv%202306.06070-b31b1b)](https://arxiv.org/abs/2306.06070) [![GitHub](https://img.shields.io/badge/GitHub-OSU--NLP--Group%2FMind2Web-181717?logo=github&logoColor=white)](https://github.com/OSU-NLP-Group/Mind2Web) [![Dataset](https://img.shields.io/badge/Dataset-HuggingFace-ff9d00?logo=huggingface&logoColor=white)](https://huggingface.co/datasets/osunlp/Mind2Web) > **Quick answer:** Mind2Web is the first dataset for building and evaluating generalist web agents, introduced by Deng et al. (Ohio State University, NeurIPS 2023 Datasets & Benchmarks track). It contains 2,350 crowdsourced tasks across 137 real-world websites and 31 domains, and tests whether an agent can generalize across unseen tasks, websites, and entire domains. The best published result — MindAct with a fine-tuned Flan-T5-XL — reaches 52.0% step success rate in-domain (Cross-Task), dropping to 38.9%/39.6% when generalizing to unseen websites/domains. ![Sample tasks and domains covered by Mind2Web](https://arxiv.org/html/2306.06070v3/x1.png) ## At a Glance **What it tests:** Whether a web agent can follow a high-level natural-language instruction and execute the correct sequence of clicks, typing, and selections on *real*, unmodified websites — and whether that ability transfers to tasks, websites, and domains never seen during training. **Why it matters:** Most earlier web-agent datasets used simulated or heavily simplified websites. Mind2Web instead captures full HTML snapshots, DOM trees, and interaction traces from 137 live, production websites across 31 domains, making it one of the most realistic benchmarks for evaluating whether an LLM-based agent can actually operate the modern web. **Known limitations:** Evaluation runs against cached offline webpage snapshots rather than live sites, so an agent's action is only credited if it matches (or is heuristically equivalent to) an action captured during data collection — a valid alternative path can be scored as a failure. GPT-4 was only evaluated on a 50-task subset per split due to API cost at the time (2023), not the full test sets. ## What Mind2Web Measures Mind2Web contains 2,350 tasks collected from 137 websites spanning 31 domains (travel, shopping, entertainment, information, and services), selected using SimilarWeb popularity rankings. Each task pairs a natural-language, high-level goal (e.g. "Book a roundtrip from Mumbai to London for two adults and a 12-year-old in premium economy") with a human-demonstrated action sequence, plus full webpage snapshots (MHTML, DOM tree, screenshots, and network traffic) captured via a custom Playwright-based annotation tool. Tasks average 7.3 actions each, and each webpage averages over 1,100 raw DOM elements — far larger and messier than the simplified sandboxes used by earlier web-agent datasets like MiniWoB++ or WebShop. The benchmark is explicitly designed to test three levels of generalization by splitting the 2,350 tasks into a 1,009-task training set and three disjoint test sets: **Cross-Task** (252 tasks, same websites seen in training, new tasks), **Cross-Website** (177 tasks, unseen websites within a seen domain), and **Cross-Domain** (912 tasks, entirely unseen domains, e.g. holding out the Information and Service top-level domains). This design directly measures whether an agent's web-navigation ability generalizes beyond the exact sites and tasks it was trained on — the central open problem the authors identify. To make raw HTML tractable for LLMs, Mind2Web's authors also introduce **MindAct**, a two-stage method: a small fine-tuned DeBERTa-v3-base ranking model first filters a webpage's ~1,100 elements down to a manageable candidate pool, then a larger LM (Flan-T5 or GPT-3.5/GPT-4 via in-context learning) selects the correct element and operation from that pool using a multi-choice question-answering format rather than free-form generation. MindAct substantially outperforms a direct-classification baseline (DeBERTa alone) and a direct-generation baseline (Flan-T5-base generating the full target span), establishing the candidate-filtering + multi-choice pattern still used by later web-agent methods such as Synapse and Agent Workflow Memory. ![The MindAct pipeline: a small ranking LM filters candidate elements, then a large LM predicts the action](https://arxiv.org/html/2306.06070v3/x3.png) ## Benchmark Specifications | Field | Value | |---|---| | Task category | Agent (web navigation) | | Metric | Step Success Rate (primary); also Element Accuracy, Operation F1, task-level Success Rate | | Number of tasks | 2,350 total — 1,009 train / 252 Cross-Task test / 177 Cross-Website test / 912 Cross-Domain test | | Websites / domains | 137 real-world websites across 31 domains | | Avg. actions per task | 7.3 | | Avg. elements per page | 1,135 (raw), 580 after cleaning | | Saturation | Low (best Cross-Task step SR: 52.0%; task-level SR in the low single digits) | | Created by | Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, Yu Su (Ohio State University) | | Source paper | [Deng et al., "Mind2Web: Towards a Generalist Agent for the Web" (arXiv 2306.06070, NeurIPS 2023)](https://arxiv.org/abs/2306.06070) | | GitHub | [OSU-NLP-Group/Mind2Web](https://github.com/OSU-NLP-Group/Mind2Web) (MIT License) | | Dataset | [osunlp/Mind2Web](https://huggingface.co/datasets/osunlp/Mind2Web) (CC BY 4.0; test set is encrypted to prevent leakage) | ## How Mind2Web Is Scored Each step of a task is evaluated independently, with the correct action history provided as context (so one wrong step doesn't cascade into automatic failure on later steps). Two component metrics are computed per step: **Element Accuracy** (did the agent select an acceptable target element, allowing for heuristically-equivalent elements) and **Operation F1** (token-level F1 between the predicted and ground-truth operation, e.g. the typed text for a `Type` action). A step counts as a **Step Success** only if both the element and the operation are correct. **Step Success Rate** is then macro-averaged across all steps in all tasks — this is the headline metric Benchgen tracks on the leaderboard below. The strictest metric, task-level **Success Rate**, requires every single step in a task to succeed — since even a 90%-accurate agent will typically fail at least one step across a 7-step task, task-level SR remains in the low single digits for every published system, underscoring how far current agents are from reliably completing full, real-world web tasks end-to-end. ## Mind2Web Leaderboard (MindAct Method) The original paper reports Step Success Rate for the MindAct method across five backbone models, on all three generalization splits (full test sets; GPT-4 uses a 50-task-per-split subset due to 2023-era API cost): | Model (via MindAct) | Cross-Task | Cross-Website | Cross-Domain | |---|---|---|---| | [Flan-T5 Base](/models/google/flan-t5-base) | 41.0% | 29.5% | 31.6% | | [Flan-T5 Large](/models/google/flan-t5-large) | 50.3% | 35.3% | 37.3% | | [Flan-T5 XL](/models/google/flan-t5-xl) | 52.0% | 38.9% | 39.6% | | [GPT-3.5 Turbo](/models/openai/gpt-3-5-turbo) (3-shot ICL) | 17.4% | 16.2% | 18.6% | | [GPT-4](/models/openai/gpt-4-0613) (3-shot ICL, 50-task subset) | 36.2% | 30.1% | 26.4% | Two non-MindAct baselines are also reported in the paper: a **Classification** baseline (the DeBERTa-v3-base candidate-ranking model used directly for element selection, with no action-prediction capability) and a **Generation** baseline (Flan-T5-base free-form generating the full target element span). Both underperform MindAct's multi-choice formulation substantially, confirming the paper's central finding that reframing element selection as multi-choice QA — rather than classification or generation — is what makes LLM-based web agents workable at this scale. For GPT-4 specifically, a later reproduction by Wang et al. (2024, "Agent Workflow Memory") ran the full MindAct + GPT-4 baseline across all three splits and additionally reported the full Element Accuracy / Operation F1 breakdown: 41.6 / 60.6 / 36.2 / 2.0 (Cross-Task), 35.8 / 51.1 / 30.1 / 2.0 (Cross-Website), and 21.6 / 52.8 / 18.6 / 1.0 (Cross-Domain), for Element Accuracy / Operation F1 / Step SR / task-level SR respectively — illustrating just how low task-level Success Rate remains even for the strongest model. ![Step success rate by website, split by Cross-Task, Cross-Website, and Cross-Domain test sets](https://arxiv.org/html/2306.06070v3/extracted/5284178/figures/website_results.png) ## Mind2Web on Benchgen Benchgen tracks 5 MindAct submissions for Mind2Web so far (Flan-T5 Base/Large/XL, GPT-3.5 Turbo, GPT-4), scored on the Cross-Task split's step success rate. [See the live leaderboard](/benchmarks/ohio-state-university/mind2web) or [submit your own model/harness](/benchmarks/ohio-state-university/mind2web). ## Mind2Web vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **Mind2Web** | **Real-website task completion across cross-task/website/domain generalization** | **2,350** | **Low** | | [GAIA2](/benchmarks/meta/gaia2) | Dynamic, time-aware, multi-agent scenarios | 800 | Low | | [ScreenSpot-Pro](/benchmarks/li-et-al/screenspot-pro) | GUI element grounding on professional software | — | Low | | [WildClawBench](/benchmarks/internlm/wildclawbench) | Real-world agentic coding/tool-use tasks | 60 | Low | Mind2Web is the earliest and still one of the most widely-cited benchmarks specifically for *web* navigation generalization — its Cross-Task/Website/Domain split design has directly influenced how later agent benchmarks (including GAIA2's capability-split evaluation) structure their generalization tests. It's the right choice for evaluating whether a model plus scaffolding can ground language instructions into concrete DOM-level actions on unmodified, real websites, as opposed to GUI-pixel grounding (ScreenSpot-Pro) or broader tool-use agents (WildClawBench). ## Run Mind2Web on Your Model Benchgen lets teams evaluate their own model and harness (candidate-filtering strategy, prompt format, multi-choice grouping) against Mind2Web's three generalization splits, tracking Step Success Rate and task-level Success Rate over time — useful for catching regressions in cross-website or cross-domain generalization introduced by a harness or prompting change, rather than relying on a single vendor-reported snapshot from 2023. ## Related Models & Benchmarks - Top performers: [Flan-T5 XL](/models/google/flan-t5-xl) · [GPT-4](/models/openai/gpt-4-0613) · [Flan-T5 Large](/models/google/flan-t5-large) - Related benchmarks: [GAIA2](/benchmarks/meta/gaia2) · [ScreenSpot-Pro](/benchmarks/li-et-al/screenspot-pro) · [WildClawBench](/benchmarks/internlm/wildclawbench) ## Frequently Asked Questions

What is Mind2Web?

Mind2Web is a dataset and benchmark introduced by Deng et al. at Ohio State University (NeurIPS 2023) for developing and evaluating generalist web agents. It contains 2,350 tasks crowdsourced from 137 real-world websites across 31 domains, each paired with a human-demonstrated action sequence and full webpage snapshots.

What does a good Mind2Web score look like?

The best published Step Success Rate is 52.0% (MindAct with fine-tuned Flan-T5-XL) on the in-domain Cross-Task split, dropping to 38.9%/39.6% on the harder Cross-Website/Cross-Domain splits. Task-level Success Rate — requiring every step of a task to succeed — remains in the low single digits (under 5%) for every published system, reflecting how demanding the full-task metric is.

Who created Mind2Web?

Mind2Web was created by Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su at The Ohio State University, and published at NeurIPS 2023's Datasets and Benchmarks track (arXiv:2306.06070).

What is MindAct, and why does Mind2Web report it instead of raw model scores?

MindAct is the two-stage method proposed alongside Mind2Web: a small fine-tuned LM (DeBERTa-v3-base) filters a webpage's ~1,100 elements down to a candidate pool, and a larger LM (Flan-T5 or GPT-3.5/GPT-4) selects the correct element and operation from that pool as a multi-choice QA problem. Because raw webpages are too large to fit in an LLM's context directly, Mind2Web's leaderboard reports harness-model combinations (MindAct + backbone) rather than base model scores alone.

Is Mind2Web saturated?

No. Even the strongest reported configuration reaches only 52.0% step success rate in the easiest (Cross-Task) setting, and task-level Success Rate — requiring a perfect run across all steps — stays below 5% for every method tested. Cross-Website and Cross-Domain generalization remain substantially harder still, roughly 10+ points lower on average.

--- *Benchmark definition based on Deng et al., "Mind2Web: Towards a Generalist Agent for the Web" (arXiv:2306.06070, NeurIPS 2023, Ohio State University). Last updated 2026-08-11.*

Rank	Model	Score
1	flan-t5-xl	52
2	flan-t5-large	50.3
3	flan-t5-base	41
4	gpt-4-0613	36.2
5	gpt-3-5-turbo	17.4

Mind2Web — Results

Mind2Web — Evaluations