Mind2Web

Mind2Web

Quick answer: Mind2Web is the first dataset for building and evaluating generalist web agents, introduced by Deng et al. (Ohio State University, NeurIPS 2023 Datasets & Benchmarks track). It contains 2,350 crowdsourced tasks across 137 real-world websites and 31 domains, and tests whether an agent can generalize across unseen tasks, websites, and entire domains. The best published result — MindAct with a fine-tuned Flan-T5-XL — reaches 52.0% step success rate in-domain (Cross-Task), dropping to 38.9%/39.6% when generalizing to unseen websites/domains.

At a Glance

What it tests: Whether a web agent can follow a high-level natural-language instruction and execute the correct sequence of clicks, typing, and selections on real, unmodified websites — and whether that ability transfers to tasks, websites, and domains never seen during training.

Why it matters: Most earlier web-agent datasets used simulated or heavily simplified websites. Mind2Web instead captures full HTML snapshots, DOM trees, and interaction traces from 137 live, production websites across 31 domains, making it one of the most realistic benchmarks for evaluating whether an LLM-based agent can actually operate the modern web.

Known limitations: Evaluation runs against cached offline webpage snapshots rather than live sites, so an agent's action is only credited if it matches (or is heuristically equivalent to) an action captured during data collection — a valid alternative path can be scored as a failure. GPT-4 was only evaluated on a 50-task subset per split due to API cost at the time (2023), not the full test sets.

What Mind2Web Measures

Mind2Web contains 2,350 tasks collected from 137 websites spanning 31 domains (travel, shopping, entertainment, information, and services), selected using SimilarWeb popularity rankings. Each task pairs a natural-language, high-level goal (e.g. "Book a roundtrip from Mumbai to London for two adults and a 12-year-old in premium economy") with a human-demonstrated action sequence, plus full webpage snapshots (MHTML, DOM tree, screenshots, and network traffic) captured via a custom Playwright-based annotation tool. Tasks average 7.3 actions each, and each webpage averages over 1,100 raw DOM elements — far larger and messier than the simplified sandboxes used by earlier web-agent datasets like MiniWoB++ or WebShop.

The benchmark is explicitly designed to test three levels of generalization by splitting the 2,350 tasks into a 1,009-task training set and three disjoint test sets: Cross-Task (252 tasks, same websites seen in training, new tasks), Cross-Website (177 tasks, unseen websites within a seen domain), and Cross-Domain (912 tasks, entirely unseen domains, e.g. holding out the Information and Service top-level domains). This design directly measures whether an agent's web-navigation ability generalizes beyond the exact sites and tasks it was trained on — the central open problem the authors identify.

To make raw HTML tractable for LLMs, Mind2Web's authors also introduce MindAct, a two-stage method: a small fine-tuned DeBERTa-v3-base ranking model first filters a webpage's ~1,100 elements down to a manageable candidate pool, then a larger LM (Flan-T5 or GPT-3.5/GPT-4 via in-context learning) selects the correct element and operation from that pool using a multi-choice question-answering format rather than free-form generation. MindAct substantially outperforms a direct-classification baseline (DeBERTa alone) and a direct-generation baseline (Flan-T5-base generating the full target span), establishing the candidate-filtering + multi-choice pattern still used by later web-agent methods such as Synapse and Agent Workflow Memory.

Benchmark Specifications

How Mind2Web Is Scored

Each step of a task is evaluated independently, with the correct action history provided as context (so one wrong step doesn't cascade into automatic failure on later steps). Two component metrics are computed per step: Element Accuracy (did the agent select an acceptable target element, allowing for heuristically-equivalent elements) and Operation F1 (token-level F1 between the predicted and ground-truth operation, e.g. the typed text for a Type action). A step counts as a Step Success only if both the element and the operation are correct. Step Success Rate is then macro-averaged across all steps in all tasks — this is the headline metric Benchgen tracks on the leaderboard below.

The strictest metric, task-level Success Rate, requires every single step in a task to succeed — since even a 90%-accurate agent will typically fail at least one step across a 7-step task, task-level SR remains in the low single digits for every published system, underscoring how far current agents are from reliably completing full, real-world web tasks end-to-end.

Mind2Web Leaderboard (MindAct Method)

The original paper reports Step Success Rate for the MindAct method across five backbone models, on all three generalization splits (full test sets; GPT-4 uses a 50-task-per-split subset due to 2023-era API cost):

Two non-MindAct baselines are also reported in the paper: a Classification baseline (the DeBERTa-v3-base candidate-ranking model used directly for element selection, with no action-prediction capability) and a Generation baseline (Flan-T5-base free-form generating the full target element span). Both underperform MindAct's multi-choice formulation substantially, confirming the paper's central finding that reframing element selection as multi-choice QA — rather than classification or generation — is what makes LLM-based web agents workable at this scale.

For GPT-4 specifically, a later reproduction by Wang et al. (2024, "Agent Workflow Memory") ran the full MindAct + GPT-4 baseline across all three splits and additionally reported the full Element Accuracy / Operation F1 breakdown: 41.6 / 60.6 / 36.2 / 2.0 (Cross-Task), 35.8 / 51.1 / 30.1 / 2.0 (Cross-Website), and 21.6 / 52.8 / 18.6 / 1.0 (Cross-Domain), for Element Accuracy / Operation F1 / Step SR / task-level SR respectively — illustrating just how low task-level Success Rate remains even for the strongest model.

Mind2Web on Benchgen

Benchgen tracks 5 MindAct submissions for Mind2Web so far (Flan-T5 Base/Large/XL, GPT-3.5 Turbo, GPT-4), scored on the Cross-Task split's step success rate. See the live leaderboard or submit your own model/harness.

Mind2Web vs Other Benchmarks

Mind2Web is the earliest and still one of the most widely-cited benchmarks specifically for web navigation generalization — its Cross-Task/Website/Domain split design has directly influenced how later agent benchmarks (including GAIA2's capability-split evaluation) structure their generalization tests. It's the right choice for evaluating whether a model plus scaffolding can ground language instructions into concrete DOM-level actions on unmodified, real websites, as opposed to GUI-pixel grounding (ScreenSpot-Pro) or broader tool-use agents (WildClawBench).

Run Mind2Web on Your Model

Benchgen lets teams evaluate their own model and harness (candidate-filtering strategy, prompt format, multi-choice grouping) against Mind2Web's three generalization splits, tracking Step Success Rate and task-level Success Rate over time — useful for catching regressions in cross-website or cross-domain generalization introduced by a harness or prompting change, rather than relying on a single vendor-reported snapshot from 2023.

Related Models & Benchmarks

• Top performers: Flan-T5 XL · GPT-4 · Flan-T5 Large
• Related benchmarks: GAIA2 · ScreenSpot-Pro · WildClawBench

Frequently Asked Questions

Benchmark definition based on Deng et al., "Mind2Web: Towards a Generalist Agent for the Web" (arXiv:2306.06070, NeurIPS 2023, Ohio State University). Last updated 2026-08-11.