SimpleQA

SimpleQA

Quick answer: SimpleQA is OpenAI's short-form factuality benchmark (Wei et al., 2024) consisting of 4,326 fact-seeking questions, each with a single indisputable answer verified by two independent human annotators. It tests whether a model knows what it knows — hallucinating a wrong answer counts worse than admitting uncertainty. Gemini 3.1 Pro leads the Inkling comparison set at 77.3%.

At a Glance

What it tests: A model's ability to accurately recall short, specific facts — names, dates, numbers, places — without hallucinating. Questions were deliberately chosen to induce hallucinations in GPT-4o and GPT-3.5, making them non-trivial for frontier models.

Why it matters: Long-form evaluations make factuality hard to measure because a response may contain dozens of claims. SimpleQA narrows the scope to single-fact questions with verifiable ground truth, making hallucination rates directly measurable and comparable across models. It was designed as a successor to saturated factuality benchmarks like TriviaQA and NQ.

Known limitations: Scope is intentionally narrow — short-form factual recall does not capture hallucinations in long-form generation. Answers must be time-stable (no questions where the correct answer changes over time), which biases toward historical facts. Grading is done by a prompted LLM classifier, introducing a small error rate.

What SimpleQA Measures

SimpleQA contains 4,326 short, fact-seeking questions across a wide range of topics — science, history, sports, entertainment, geography, and more. Every question has exactly one correct, time-stable answer and was verified by two independent human annotators; only questions where both annotators agreed were included. A random sample of 1,000 questions showed a 94.4% human accuracy rate, with an estimated ~3% inherent error rate in the dataset.

Grading is done by a prompted LLM classifier (typically GPT-4o) that sees both the model's predicted answer and the ground-truth answer and returns one of three verdicts:

• Correct — the predicted answer fully contains the ground-truth answer without contradiction
• Incorrect — the predicted answer contradicts the ground truth in any way, even if hedged
• Not attempted — the model declined to answer without contradicting the reference

The key insight is that "not attempted" is better than "incorrect": a model that says "I don't know" is more trustworthy than one that confidently gives a wrong answer. Scores are typically reported as % correct.

Benchmark Specifications

How SimpleQA Is Scored

Each question is answered by the model in a zero-shot setting. A grading model (GPT-4o or equivalent) then classifies the response as correct, incorrect, or not attempted. Scores are reported as % correct. Some evaluations also report the "not attempted" rate separately as a measure of calibration.

Thinking/reasoning models tend to have higher "not attempted" rates than standard chat models because they can recognize uncertainty — this is generally considered a feature, not a bug.

State-of-the-Art Results

Scores from Inkling model card (Thinking Machines Lab, July 2026), evaluated at effort=0.99.

The gap between closed-weight models (68–77%) and open-weight models (32–57%) on SimpleQA reflects the larger parametric knowledge capacity of closed frontier models.

SimpleQA vs Related Benchmarks

Last updated 2026-07-16.