# Natural Questions ![Category](https://img.shields.io/badge/Category-Open%20Domain%20QA-1f6feb) ![Metric](https://img.shields.io/badge/Metric-Long%2FShort%20F1-2da44e) ![Saturation](https://img.shields.io/badge/Saturation-Medium-d29922) ![Created](https://img.shields.io/badge/Created-2019-555) [![GitHub](https://img.shields.io/badge/GitHub-google--research--datasets%2Fnatural--questions-181717?logo=github&logoColor=white)](https://github.com/google-research-datasets/natural-questions) [![Dataset](https://img.shields.io/badge/Dataset-HuggingFace-ff9d00?logo=huggingface&logoColor=white)](https://huggingface.co/datasets/google-research-datasets/natural_questions) > **Quick answer:** Natural Questions (NQ) is Google Research's open-domain question-answering benchmark built from real anonymized Google search queries, each paired with a full Wikipedia page as evidence, created by Kwiatkowski et al. (2019). The official blind-test long/short-answer leaderboard doesn't expose recent frontier-model rows publicly; the best figures currently available come from a third-party aggregator using a different normalized metric — treat these as directional, not authoritative. --- ## At a Glance **What it tests:** Whether a system can find the right answer inside a long Wikipedia page for a real user search query, including correctly recognizing when no answer is present. **Why it matters:** NQ's queries are genuine Google search traffic rather than crowd-written questions, making it more representative of real information-seeking behavior than many earlier QA datasets. **Known limitations:** Benchgen could not verify current frontier-model scores against the official long/short-answer F1 blind-test leaderboard — the table below uses a third-party aggregator's normalized 0–1 metric on a small set of smaller open models, and should be read with that caveat in mind. ## What Natural Questions Measures NQ pairs real, anonymized Google search queries with full Wikipedia pages and asks a system to produce a long answer (the passage containing the answer) and a short answer (the specific span, or "no answer" if none exists). This two-level structure — plus evidence coming from full, unedited Wikipedia pages rather than short curated passages — makes NQ more realistic than earlier open-domain QA datasets, since a system has to locate relevant evidence inside a long document rather than a hand-picked paragraph. ## Benchmark Specifications | Field | Value | |---|---| | Training examples | 307,373 | | Dev examples | 7,830 | | Test examples | 7,842 | | Task category | Open-domain QA | | Metric | Long-answer and short-answer precision / recall / F1 | | Released | 2019 | | Created by | Tom Kwiatkowski, Jennimaria Palomaki, and collaborators (Google Research) | ## How Natural Questions Is Scored Systems must produce a long answer (passage-level span) and a short answer (exact span, or explicitly "no answer") for each query; both are scored against human-annotated ground truth using precision, recall, and F1. The official leaderboard uses a hidden test set to prevent overfitting to public annotations. ## Natural Questions on Benchgen > No Benchgen results yet — [be the first to run Natural Questions](/benchmarks/natural-questions). ## Natural Questions vs Other Benchmarks | Benchmark | What it tests | Saturation | |---|---|---| | Natural Questions | Open-domain QA from real Google search queries | Medium | | [TriviaQA](/benchmarks/triviaqa) | Open-domain reading comprehension from trivia + evidence | High | | [DROP](/benchmarks/drop) | Discrete reasoning over paragraphs | — | ## Run Natural Questions on Your Model Benchgen lets you run Natural Questions against your own model with version-controlled, regression-tracked results — a useful way to get a verified score given how sparse public frontier-model data currently is for this benchmark. ## Related Models & Benchmarks - [TriviaQA](/benchmarks/triviaqa) - [DROP](/benchmarks/drop) ## Frequently Asked Questions

What is Natural Questions?

Natural Questions is Google Research's open-domain QA benchmark built from real anonymized search queries paired with full Wikipedia pages as evidence, published in 2019.

What does a good score look like?

On the official long/short-answer F1 metric, strong systems have historically scored in the 60-70s; current frontier-LLM figures on the official blind test aren't publicly available, so treat any third-party aggregator score as directional only.

Who created Natural Questions?

Natural Questions was created by Tom Kwiatkowski, Jennimaria Palomaki, Michael Collins, and collaborators at Google Research, published in 2019.

Rank	Model	Score
1	gemma-2-27b	0.345
2	mistral-nemo-instruct	0.312
3	gemma-2-9b	0.292
4	gemma-3n-e4b-litert-preview	0.209
5	gemma-3n-e4b	0.209
6	gemma-3n-e2b-litert-preview	0.155
7	gemma-3n-e2b	0.155

Natural Questions — Results

Natural Questions — Evaluations