# ZeroBench ![Category](https://img.shields.io/badge/Category-Visual%20Reasoning-1f6feb) ![Metric](https://img.shields.io/badge/Metric-%25%20Accuracy-2da44e) ![Questions](https://img.shields.io/badge/Questions-100%20%2B%20334%20sub--questions-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-Very%20Low-d29922) ![Created](https://img.shields.io/badge/Created-Feb%202025-555) [![Paper](https://img.shields.io/badge/Paper-arXiv%202502.09696-b31b1b)](https://arxiv.org/abs/2502.09696) > **Quick answer:** ZeroBench (Roberts et al., ICML 2026) is a lightweight, deliberately extremely challenging visual reasoning benchmark consisting of 100 curated, natural, and unambiguous questions along with 334 subquestions, designed so that current frontier multimodal models score 0% at release — hence the name. Kimi K3 scores 41.0% (pass@5) as of July 2026, a substantial jump from initial near-zero scores. ## At a Glance **What it tests:** Complex, multi-step visual reasoning that current frontier vision-language models cannot solve at all on first release — designed to leave enormous headroom for future progress. **Why it matters:** Most vision benchmarks saturate within a year or two of release. ZeroBench was explicitly designed to remain unsaturated for years, giving researchers a durable yardstick for genuine progress in visual reasoning. **Known limitations:** Because the main task is designed to be near-impossible, most meaningful comparison currently happens on the 334 easier subquestions rather than the full 100-question main set. ## What ZeroBench Measures ZeroBench targets the hardest end of visual reasoning: interpreting dense or ambiguous visual scenes, performing multi-hop inference across visual elements, and reaching a single unambiguous answer. Every question was manually vetted to ensure it is natural (not adversarially constructed) and unambiguous, while still proving essentially unsolvable for frontier vision-language models at time of release. The accompanying 334 subquestions decompose the harder main questions into more tractable steps, providing a gradient of difficulty for tracking incremental progress. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Vision / reasoning | | Metric | % accuracy | | Main questions | 100 | | Sub-questions | 334 | | Saturation | Very low (0% at release) | | Created by | Roberts et al. | | Source paper | [ZeroBench (arXiv 2502.09696, ICML 2026)](https://arxiv.org/abs/2502.09696) | ## How ZeroBench Is Scored Models attempt each question and are scored on % accuracy against ground truth. Because the main 100-question set is designed to be extremely difficult, scores are often reported using pass@k (e.g., pass@5, allowing up to 5 sampled attempts per question) to track gradual capability gains as models improve. ## State-of-the-Art Results | Rank | Model | Score | Source | Date | |---|---|---|---|---| | 1 | [Kimi K3](/models/moonshot/kimi-k3) | 41.0% (pass@5) | [Kimi K3 technical report](https://www.kimi.com/blog/kimi-k3) | 2026-07 | *Score sourced from Moonshot AI's Kimi K3 announcement, July 2026.* ## ZeroBench on Benchgen > No Benchgen results yet — [be the first to run ZeroBench](/benchmarks/zerobench). ## ZeroBench vs Other Benchmarks | Benchmark | What it tests | Saturation | |---|---|---| | **ZeroBench** | Deliberately near-impossible visual reasoning | Very low | | [Humanity's Last Exam](/benchmarks/humanitys-last-exam) | Deliberately hardest-frontier knowledge/reasoning | Low | | [ARC-AGI v2](/benchmarks/arc-agi-v2) | Abstract visual reasoning | Low | | [MathVision](/benchmarks/mathvision) | Visual mathematical reasoning | Low | ## Run ZeroBench on Your Model Benchgen lets you run ZeroBench against your own multimodal model, tracking progress on both the main 100-question set and the 334 subquestions as your model improves over time. ## Frequently Asked Questions

What is ZeroBench?

ZeroBench is a deliberately extremely difficult visual reasoning benchmark of 100 curated questions (plus 334 subquestions), designed so frontier models scored 0% at release, leaving years of headroom for future progress.

What does a good score look like on ZeroBench?

At release in early 2025, all tested frontier models scored 0% on the main question set. By mid-2026, leading multimodal models such as Kimi K3 report pass@5 scores around 41%, reflecting rapid capability gains.

Who created ZeroBench?

ZeroBench was created by Jonathan Roberts and collaborators (arXiv:2502.09696), accepted at ICML 2026, with a project page at zerobench.github.io.

Rank	Model	Score
1	deepseek-v4-1-flash-1017	51
2	muse-spark-open-816b-a42b	50
3	deepseek-v4-1-flash	49
4	kimi-k3	41
5	deepseek-v4-flash-vision-exp	35

ZeroBench — Results

ZeroBench — Evaluations