Natural Questions — Results
| Rank | Model | Score |
|---|---|---|
| 1 | gemma-2-27b | 0.345 |
| 2 | mistral-nemo-instruct | 0.312 |
| 3 | gemma-2-9b | 0.292 |
| 4 | gemma-3n-e4b-litert-preview | 0.209 |
| 5 | gemma-3n-e4b | 0.209 |
| 6 | gemma-3n-e2b-litert-preview | 0.155 |
| 7 | gemma-3n-e2b | 0.155 |
Natural Questions
1 phaseActive
Open-domain QA benchmark built from real Google search queries and full Wikipedia pages. Metric: long/short-answer F1. Official blind test scores are limited.
Overview
Natural Questions
Quick answer: Natural Questions (NQ) is Google Research's open-domain question-answering benchmark built from real anonymized Google search queries, each paired with a full Wikipedia page as evidence, created by Kwiatkowski et al. (2019). The official blind-test long/short-answer leaderboard doesn't expose recent frontier-model rows publicly; the best figures currently available come from a third-party aggregator using a different normalized metric — treat these as directional, not authoritative.
At a Glance
What it tests: Whether a system can find the right answer inside a long Wikipedia page for a real user search query, including correctly recognizing when no answer is present. Why it matters: NQ's queries are genuine Google search traffic rather than crowd-written questions, making it more representative of real information-seeking behavior than many earlier QA datasets. Known limitations: Benchgen could not verify current frontier-model scores against the official long/short-answer F1 blind-test leaderboard — the table below uses a third-party aggregator's normalized 0–1 metric on a small set of smaller open models, and should be read with that caveat in mind.
What Natural Questions Measures
NQ pairs real, anonymized Google search queries with full Wikipedia pages and asks a system to produce a long answer (the passage containing the answer) and a short answer (the specific span, or "no answer" if none exists). This two-level structure — plus evidence coming from full, unedited Wikipedia pages rather than short curated passages — makes NQ more realistic than earlier open-domain QA datasets, since a system has to locate relevant evidence inside a long document rather than a hand-picked paragraph.
Benchmark Specifications
| Field | Value |
|---|---|
| Training examples | 307,373 |
| Dev examples | 7,830 |
| Test examples | 7,842 |
| Task category | Open-domain QA |
| Metric | Long-answer and short-answer precision / recall / F1 |
| Released | 2019 |
| Created by | Tom Kwiatkowski, Jennimaria Palomaki, and collaborators (Google Research) |
How Natural Questions Is Scored
Systems must produce a long answer (passage-level span) and a short answer (exact span, or explicitly "no answer") for each query; both are scored against human-annotated ground truth using precision, recall, and F1. The official leaderboard uses a hidden test set to prevent overfitting to public annotations.
Natural Questions on Benchgen
No Benchgen results yet — be the first to run Natural Questions.
Natural Questions vs Other Benchmarks
| Benchmark | What it tests | Saturation |
|---|---|---|
| Natural Questions | Open-domain QA from real Google search queries | Medium |
| TriviaQA | Open-domain reading comprehension from trivia + evidence | High |
| DROP | Discrete reasoning over paragraphs | — |
Run Natural Questions on Your Model
Benchgen lets you run Natural Questions against your own model with version-controlled, regression-tracked results — a useful way to get a verified score given how sparse public frontier-model data currently is for this benchmark.