VisualWebBench
1 phaseActive
Seven-task multimodal web page understanding and grounding benchmark across 139 real websites, created by CMU, CUHK, and collaborating researchers.
Overview
VisualWebBench
Quick answer: VisualWebBench is a multimodal benchmark that tests how well MLLMs understand and ground content on real web pages, covering seven tasks — from OCR and captioning to element grounding and action prediction — across 1.5K human-curated instances from 139 real websites.
At a Glance
What it tests: Fine-grained web-page perception and grounding abilities (as opposed to end-to-end web-agent task completion), including webpage captioning, OCR, element/action grounding, and webpage QA.
Why it matters: Prior benchmarks either evaluated general multimodal tasks that didn't capture web-specific characteristics, or evaluated end-to-end web agents without isolating which sub-skill (OCR, grounding, layout understanding) was failing — VisualWebBench decomposes this into seven measurable tasks.
Known limitations: As with most GUI/web benchmarks, coverage is limited to a curated snapshot of real sites (139 websites, 87 sub-domains) and does not test multi-step task completion or dynamic page interaction.
What VisualWebBench Measures
VisualWebBench evaluates MLLMs across seven tasks spanning three broad categories: web-page-level tasks (captioning, webpage QA), element-level tasks (element OCR, element grounding), and action-level tasks (action grounding, action prediction, heading OCR). The dataset comprises 1.5K human-curated instances drawn from 139 real websites across 87 sub-domains, deliberately built to capture web-specific characteristics that general multimodal benchmarks miss — dense text, small UI elements, and structured layouts.
The paper's evaluation of 14 open-source MLLMs plus Gemini Pro, the Claude-3 series, and GPT-4V(ision) found significant performance gaps, especially in grounding within text-rich environments and processing lower-resolution inputs — highlighting web understanding as a distinct capability from general image understanding.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Multimodal (web page understanding and grounding) |
| Metric | Task-specific accuracy, F1, or IoU-based grounding score across the 7 tasks |
| Number of tasks | 1,500 human-curated instances across 7 tasks, 139 websites, 87 sub-domains |
| Saturation | Medium — frontier proprietary models lead but still show clear gaps versus human performance, especially on grounding tasks |
| Created by | Junpeng Liu, Yifan Song, Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li, Xiang Yue |
| Source paper | Liu et al. 2024 |
| Dataset | Hugging Face · GitHub |
How VisualWebBench Is Scored
Each of the seven tasks uses a metric suited to its output format: captioning and QA tasks use text-similarity or accuracy metrics against reference answers; OCR tasks use text-matching accuracy; grounding tasks (element and action grounding) use accuracy based on whether the predicted bounding box or coordinate falls within the target region. An overall score averages normalized performance across all seven tasks.
State-of-the-Art Results
Scores fluctuate by model release cadence and evaluation harness (prompt template, resolution handling). See the official leaderboard for current standings across all 7 tasks.
VisualWebBench on Benchgen
No Benchgen results yet — be the first to run VisualWebBench.
VisualWebBench vs Other Benchmarks
| Benchmark | What it tests | Tasks | Saturation |
|---|---|---|---|
| VisualWebBench | Fine-grained web page understanding + grounding (7 tasks) | 1,500 | medium |
| ScreenSpot | Single-step GUI element click grounding | — | medium |
| MM-Mind2Web | End-to-end multi-step web agent task completion | — | low |
VisualWebBench's fine-grained task decomposition complements end-to-end web-agent benchmarks like MM-Mind2Web, helping isolate whether agent failures stem from perception/grounding gaps or planning gaps.
Run VisualWebBench on Your Model
Benchgen lets teams run all seven VisualWebBench tasks against their own model versions on demand, with per-task breakdowns to pinpoint whether OCR, grounding, or QA is the weakest link for web-focused use cases.
Related Models & Benchmarks
Explore related web and GUI benchmarks like ScreenSpot and MM-Mind2Web to compare fine-grained perception and end-to-end agentic evaluation approaches.