Benchgen

SimpleQA Verified — Results

RankModelScore
1ling-3-0-flash-base26.9
2ling-3-0-tiny-base7.2
S

SimpleQA Verified

1 phaseActive

Google DeepMind's curated, de-duplicated, topic-balanced 1,000-prompt refinement of OpenAI's SimpleQA factuality benchmark.

Overview

SimpleQA Verified

Category Metric Saturation Created

Paper

Quick answer: SimpleQA Verified (Haas et al., 2025, Google DeepMind) is a curated, de-duplicated, topic-balanced 1,000-prompt refinement of OpenAI's SimpleQA factuality benchmark. InclusionAI's Ling-3.0-flash-base scores 26.90% correct as of 2026.

At a Glance

What it tests: Whether a model can answer short, fact-seeking questions correctly from parametric knowledge alone, using a cleaned-up, de-duplicated, and topic-balanced subset of the original SimpleQA question pool.

Why it matters: OpenAI's original SimpleQA has known data-quality issues (duplicate and ambiguous questions); SimpleQA Verified was built by Google DeepMind to provide a more reliable short-form factuality signal while remaining comparable in spirit to the original benchmark.

Known limitations: SimpleQA Verified is a distinct, harder-curated dataset from the original simpleqa.md benchmark on Benchgen — scores on the two are not directly comparable. Scores cited here are self-reported by model developers rather than pulled from a shared third-party leaderboard.

What SimpleQA Verified Measures

SimpleQA Verified evaluates a model's ability to answer short, fact-seeking questions with a single indisputable answer, drawing on a curated, de-duplicated, and topic-balanced 1,000-prompt subset derived from OpenAI's SimpleQA to give a more reliable factuality signal.

Benchmark Specifications

FieldValue
Task categoryReasoning (parametric factual knowledge)
Metric% correct
Number of questions1,000
Created byHaas et al.
OrgGoogle DeepMind
Source paperHaas et al. 2025
Public leaderboardKaggle

How SimpleQA Verified Is Scored

Models are scored on the percentage of the 1,000 curated questions answered correctly, graded against each question's single indisputable answer.

State-of-the-Art Results

Scores above are self-reported by InclusionAI on their respective model cards and shown for context. They are not Benchgen measurements, and are not drawn from a shared third-party leaderboard — only the 2 scores above are currently tracked locally.

SimpleQA Verified on Benchgen

No Benchgen results yet — be the first to run SimpleQA Verified.

SimpleQA Verified vs Other Benchmarks

BenchmarkWhat it testsSaturation
SimpleQA VerifiedCurated, de-duplicated short-form factualityLow
SimpleQAOriginal short-form factuality (OpenAI)Medium

SimpleQA Verified is distinct from the original SimpleQA in that its question pool has been de-duplicated and topic-balanced by Google DeepMind, making it a stricter, non-interchangeable factuality signal rather than a drop-in replacement.

Run SimpleQA Verified on Your Model

Benchgen lets you evaluate your own model against SimpleQA Verified's curated factuality question set and track results across model versions.

Frequently Asked Questions

What is SimpleQA Verified? SimpleQA Verified is a curated, de-duplicated, topic-balanced 1,000-prompt refinement of OpenAI's SimpleQA factuality benchmark, introduced by Google DeepMind (Haas et al., 2025).
What does a good SimpleQA Verified score look like? InclusionAI's Ling-3.0-flash-base reports 26.90% correct as of 2026, the highest currently tracked on Benchgen; Ling-3.0-tiny-base, a smaller sibling model, scores 7.20%.
Who created SimpleQA Verified? SimpleQA Verified was created by Google DeepMind (Haas et al.); see the original paper and its Kaggle leaderboard.
Is SimpleQA Verified the same as SimpleQA? No — SimpleQA Verified is a distinct, harder-curated variant built by Google DeepMind, not directly comparable to the original OpenAI SimpleQA benchmark's scores.

Benchmark definition paraphrased from Haas et al. 2025. State-of-the-art scores sourced from published model cards and attributed inline. Last updated 2026-11-03.