# VisualWebBench ![Category](https://img.shields.io/badge/Category-Multimodal-1f6feb) ![Metric](https://img.shields.io/badge/Metric-Accuracy%20%2F%20F1%20%2F%20IoU-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-1%2C500-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-medium-d29922) ![Created](https://img.shields.io/badge/Created-Apr%202024-555) [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/abs/2404.05955) [![GitHub](https://img.shields.io/badge/GitHub-Repo-181717?logo=github&logoColor=white)](https://github.com/VisualWebBench/VisualWebBench) > **Quick answer:** VisualWebBench is a multimodal benchmark that tests how well MLLMs understand and ground content on real web pages, covering seven tasks — from OCR and captioning to element grounding and action prediction — across 1.5K human-curated instances from 139 real websites. ## At a Glance **What it tests:** Fine-grained web-page perception and grounding abilities (as opposed to end-to-end web-agent task completion), including webpage captioning, OCR, element/action grounding, and webpage QA. **Why it matters:** Prior benchmarks either evaluated general multimodal tasks that didn't capture web-specific characteristics, or evaluated end-to-end web agents without isolating which sub-skill (OCR, grounding, layout understanding) was failing — VisualWebBench decomposes this into seven measurable tasks. **Known limitations:** As with most GUI/web benchmarks, coverage is limited to a curated snapshot of real sites (139 websites, 87 sub-domains) and does not test multi-step task completion or dynamic page interaction. ## What VisualWebBench Measures VisualWebBench evaluates MLLMs across seven tasks spanning three broad categories: **web-page-level tasks** (captioning, webpage QA), **element-level tasks** (element OCR, element grounding), and **action-level tasks** (action grounding, action prediction, heading OCR). The dataset comprises 1.5K human-curated instances drawn from 139 real websites across 87 sub-domains, deliberately built to capture web-specific characteristics that general multimodal benchmarks miss — dense text, small UI elements, and structured layouts. The paper's evaluation of 14 open-source MLLMs plus Gemini Pro, the Claude-3 series, and GPT-4V(ision) found significant performance gaps, especially in grounding within text-rich environments and processing lower-resolution inputs — highlighting web understanding as a distinct capability from general image understanding. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Multimodal (web page understanding and grounding) | | Metric | Task-specific accuracy, F1, or IoU-based grounding score across the 7 tasks | | Number of tasks | 1,500 human-curated instances across 7 tasks, 139 websites, 87 sub-domains | | Saturation | Medium — frontier proprietary models lead but still show clear gaps versus human performance, especially on grounding tasks | | Created by | Junpeng Liu, Yifan Song, Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li, Xiang Yue | | Source paper | [Liu et al. 2024](https://arxiv.org/abs/2404.05955) | | Dataset | [Hugging Face](https://huggingface.co/datasets/visualwebbench/VisualWebBench) · [GitHub](https://github.com/VisualWebBench/VisualWebBench) | ## How VisualWebBench Is Scored Each of the seven tasks uses a metric suited to its output format: captioning and QA tasks use text-similarity or accuracy metrics against reference answers; OCR tasks use text-matching accuracy; grounding tasks (element and action grounding) use accuracy based on whether the predicted bounding box or coordinate falls within the target region. An overall score averages normalized performance across all seven tasks. ## State-of-the-Art Results > Scores fluctuate by model release cadence and evaluation harness (prompt template, resolution handling). See the [official leaderboard](https://github.com/VisualWebBench/VisualWebBench) for current standings across all 7 tasks. ## VisualWebBench on Benchgen > No Benchgen results yet — [be the first to run VisualWebBench](/benchmarks/junpeng-liu-yifan-song-bill-yuchen-lin-wai-lam-graham-neubig-yuanzhi-li-xiang-yue/visualwebbench). ## VisualWebBench vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **VisualWebBench** | Fine-grained web page understanding + grounding (7 tasks) | 1,500 | medium | | [ScreenSpot](/benchmarks/screenspot) | Single-step GUI element click grounding | — | medium | | [MM-Mind2Web](/benchmarks/mm-mind2web) | End-to-end multi-step web agent task completion | — | low | VisualWebBench's fine-grained task decomposition complements end-to-end web-agent benchmarks like MM-Mind2Web, helping isolate whether agent failures stem from perception/grounding gaps or planning gaps. ## Run VisualWebBench on Your Model Benchgen lets teams run all seven VisualWebBench tasks against their own model versions on demand, with per-task breakdowns to pinpoint whether OCR, grounding, or QA is the weakest link for web-focused use cases. ## Related Models & Benchmarks Explore related web and GUI benchmarks like [ScreenSpot](/benchmarks/screenspot) and [MM-Mind2Web](/benchmarks/mm-mind2web) to compare fine-grained perception and end-to-end agentic evaluation approaches. ## Frequently Asked Questions

What is VisualWebBench?

VisualWebBench is a multimodal benchmark with 1,500 instances across 7 tasks testing web page understanding and grounding, drawn from 139 real websites spanning 87 sub-domains.

What does a good VisualWebBench score look like?

Even leading proprietary models like GPT-4V and Claude-3 show meaningful performance gaps versus expected human-level performance, particularly on element and action grounding tasks.

Who created VisualWebBench?

VisualWebBench was created by Junpeng Liu and collaborators across CUHK, CMU, and the Allen Institute for AI.

VisualWebBench — Leaderboard

VisualWebBench Leaderboard