WebArena-Verified — Results
| Rank | Model | Score |
|---|---|---|
| 1 | qwen3-8-27b | 64.8 |
WebArena-Verified
1 phaseActive
An audited, deterministically-scored re-release of the 812-task WebArena browser-use benchmark, testing autonomous agents on realistic e-commerce, social-forum, GitLab, and CMS web tasks. Metric: % task success.
Overview
WebArena-Verified
Quick answer: WebArena-Verified is an audited, deterministically-scored re-release (ServiceNow, 2025) of the original WebArena benchmark (Zhou et al., CMU, 2023) — a fully functional, self-hostable web environment with 812 realistic long-horizon tasks across e-commerce, social-forum, GitLab, and content-management sites, used to test autonomous browser-use agents. Qwen3.8-27B scores 64.8% as of August 2026.
At a Glance
What it tests: Whether an autonomous agent can complete realistic, long-horizon natural-language tasks inside a fully functional web environment spanning four site categories (e-commerce, social forum discussions, collaborative software development/GitLab, content management), with success verified deterministically rather than by fuzzy string-matching or LLM-as-judge.
Why it matters: The original 2023 WebArena paper found even strong GPT-4-based agents succeeded on only 14.4% of tasks (vs. 78.2% human performance), exposing a large gap between synthetic agent benchmarks and realistic browser use. WebArena-Verified audits every task, reference answer, and evaluator from that original release and replaces nondeterministic judging with type-aware, structural comparison, making scores more reliable and reproducible across agents and model versions.
Known limitations: The underlying task set is the same 812 tasks as the original 2023 WebArena (re-audited, not newly authored), so it inherits that benchmark's site/domain scope; a curated 258-task "Hard" subset exists for faster, lower-cost evaluation but is a different, smaller scope than the full-dataset score reported here.
What WebArena-Verified Measures
WebArena-Verified distributes a curated, version-controlled dataset of the original WebArena's 812 web tasks together with deterministic evaluators that operate on agent responses and captured network traces, enabling offline evaluation without a live web environment. It covers four realistic, self-hosted site categories — e-commerce, social forum discussions, GitLab-style collaborative development, and content management — enriched with tools (e.g. a map) and external knowledge bases (e.g. user manuals) to encourage human-like task-solving, exactly as specified in the original WebArena environment.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Agent (browser-use) |
| Metric | % task success |
| Number of tasks | 812 (full verified dataset; a 258-task "Hard" subset is also available) |
| Saturation | Low |
| Created by | Zhou et al. (original WebArena, CMU); El hattami, Thakkar, Chapados, Pal (Verified re-release, ServiceNow) |
| Source paper | Zhou et al. 2023 |
| GitHub | ServiceNow/webarena-verified |
How WebArena-Verified Is Scored
Each of the 812 tasks is scored by a deterministic, type-aware evaluator that compares an agent's response and captured network trace against a manually-reviewed reference answer — replacing the original WebArena's LLM-as-judge and substring-matching scoring with structural comparison; the overall score is the percentage of tasks completed successfully.
State-of-the-Art Results
| Rank | Model | Score | Source | Date |
|---|---|---|---|---|
| 1 | Qwen3.8-27B | 64.8% | Qwen3.8-27B model card | 2026-08 |
Score above is self-reported by Alibaba's Qwen Team on the official Hugging Face model card and shown for context. It is not a Benchgen measurement, and is not drawn from a shared third-party leaderboard — only the 1 score above is currently tracked locally.
WebArena-Verified on Benchgen
No Benchgen results yet — be the first to run WebArena-Verified.
WebArena-Verified vs Other Benchmarks
| Benchmark | What it tests | Saturation |
|---|---|---|
| WebArena-Verified | Realistic, long-horizon browser-use agent control across e-commerce/forum/GitLab/CMS sites | Low |
| AndroidWorld | Dynamic, parameterized mobile-use agent control | Low |
| OSWorld-Verified | Real desktop/web computer-use agent control | — |
WebArena-Verified is the browser-use counterpart to device-level agent benchmarks like AndroidWorld (mobile) and OSWorld-Verified (desktop) — use it to assess whether an agent can complete realistic, multi-step tasks purely within a web browser across diverse site categories, independent of mobile or desktop OS control.
Run WebArena-Verified on Your Model
Benchgen lets you evaluate your own model against WebArena-Verified's 812 audited browser-use tasks and track results across model versions.
Related Models & Benchmarks
- Top performers: Qwen3.8-27B
- Related benchmarks: AndroidWorld · OSWorld-Verified
Frequently Asked Questions
What is WebArena-Verified?
WebArena-Verified is an audited, deterministically-scored re-release (ServiceNow, 2025) of the original WebArena benchmark (Zhou et al., CMU, 2023) — a fully functional web environment with 812 realistic tasks across e-commerce, social-forum, GitLab, and content-management sites, used to evaluate autonomous browser-use agents.What does a good WebArena-Verified score look like?
Qwen3.8-27B reports 64.8% as of August 2026, the only score currently tracked on Benchgen. For context, the original 2023 paper's best GPT-4-based agent achieved only 14.4% end-to-end task success, versus 78.2% human performance — so scores in the 60%+ range reflect substantial progress since release.Who created WebArena-Verified?
The original WebArena benchmark was created by Shuyan Zhou and co-authors at Carnegie Mellon University (2023); the audited WebArena-Verified re-release was created by Amine El hattami, Megh Thakkar, Nicolas Chapados, and Christopher Pal, presented at the Scaling Environments for Agents (SEA) Workshop at NeurIPS 2025.Is WebArena-Verified saturated?
No — WebArena-Verified remains a low-saturation benchmark; its 812 realistic, long-horizon browser tasks with deterministic scoring keep meaningful headroom between models.How does WebArena-Verified differ from AndroidWorld?
WebArena-Verified tests agent control purely within a web browser across e-commerce/forum/GitLab/CMS sites, while AndroidWorld tests multi-step agent control on a real Android device across 20 mobile apps — both verify success against ground-truth state rather than static answer-matching, but for different device/platform scopes.Benchmark definition paraphrased from Zhou et al. 2023 (original WebArena) and the ServiceNow/webarena-verified re-release documentation. State-of-the-art scores sourced from published model cards and attributed inline. Last updated 2026-10-31.