WideSearch
1 phaseActive
ByteDance Seed's benchmark for agentic broad info-seeking, testing whether search agents can reliably collect and organize large-scale, verifiable information.
Overview
WideSearch
Quick answer: WideSearch is a benchmark of 200 manually curated questions (100 English, 100 Chinese) across 15+ domains that tests whether agentic search systems can reliably collect large-scale, atomic pieces of information from the web and organize them into a complete, verifiable output — a task on which most current systems succeed less than 5% of the time.
At a Glance
What it tests: Whether single-agent, multi-agent, and commercial agentic search systems can perform "wide-context" information collection — gathering many individually-verifiable facts (not just one deep answer) and assembling them correctly.
Why it matters: Many real workflows (market research, competitive analysis, trip planning) require broad, exhaustive, and accurate information-gathering rather than a single deep answer; WideSearch shows that even strong commercial search agents currently fail almost all of these tasks.
Known limitations: At 200 tasks, WideSearch is modest in scale, and because task difficulty is calibrated so that human testers can achieve near-100% accuracy with sufficient time, gains on WideSearch mainly reflect agent reliability rather than raw task difficulty for humans.
What WideSearch Measures
WideSearch's 200 questions are grounded in real user queries spanning over 15 diverse domains, and were passed through a rigorous five-stage quality-control pipeline to ensure each task's difficulty, completeness, and verifiability. Each task requires an agent to collect large-scale atomic information — individually verifiable facts — and arrange it into a well-organized final output, mirroring real-world workflows like a financial analyst compiling metrics across every company in a sector.
The paper benchmarks more than 10 state-of-the-art agentic search systems, spanning single-agent setups, multi-agent frameworks, and end-to-end commercial products. Most systems achieve overall success rates near 0%, with the best performer reaching only 5% — while cross-validated human testers, given sufficient time, can achieve close to 100% success.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Agent / broad, verifiable information-seeking |
| Metric | Task success rate (%), graded against per-item verifiable ground truth |
| Number of tasks | 200 questions (100 English, 100 Chinese) across 15+ domains |
| Saturation | Very low — best system reaches only 5% success versus near-100% for humans |
| Created by | Ryan Wong, Jiawei Wang, Junjie Zhao, Li Chen, Yan Gao, Long Zhang, Xuan Zhou, Zuo Wang, Kai Xiang, et al. (ByteDance Seed) |
| Source paper | Wong et al. 2025 |
| GitHub | ByteDance-Seed/WideSearch |
How WideSearch Is Scored
Each task's expected output is a set of atomic, individually-verifiable facts arranged in a structured format; an agent's response is graded by checking each atomic fact against ground truth, and overall success rate reflects the share of tasks where the agent's compiled output is judged complete and correct.
State-of-the-Art Results
The WideSearch paper reports that most of the more than 10 evaluated agentic search systems — including single-agent, multi-agent, and commercial end-to-end products — achieve overall success rates near 0%, with the best performer reaching just 5%. See the project page for the full per-system breakdown.
WideSearch on Benchgen
No Benchgen results yet — be the first to run WideSearch.
WideSearch vs Other Benchmarks
| Benchmark | What it tests | Tasks | Saturation |
|---|---|---|---|
| WideSearch | Broad, verifiable web information-gathering | 200 | very low |
| FRAMES | Multi-hop factual reasoning and retrieval | — | — |
| Vending-Bench 2 | Long-horizon agent coherence via simulated business | 1 (year-long run) | low |
WideSearch is distinguished from single-answer retrieval benchmarks by requiring exhaustive, breadth-first collection of many verifiable facts rather than one deep or narrow answer, exposing a distinct failure mode current search agents largely have not solved.
Run WideSearch on Your Model
Benchgen lets teams evaluate their own agent against WideSearch's broad information-gathering tasks, surfacing whether it can reliably compile complete, verifiable results rather than partial or hallucinated ones.