# TriviaQA ![Category](https://img.shields.io/badge/Category-Reading%20Comprehension-1f6feb) ![Metric](https://img.shields.io/badge/Metric-EM%20%2F%20F1-2da44e) ![Saturation](https://img.shields.io/badge/Saturation-High-red) ![Created](https://img.shields.io/badge/Created-May%202017-555) [![Paper](https://img.shields.io/badge/Paper-arXiv%201705.03551-b31b1b)](https://arxiv.org/abs/1705.03551) [![GitHub](https://img.shields.io/badge/GitHub-mandarjoshi90%2Ftriviaqa-181717?logo=github&logoColor=white)](https://github.com/mandarjoshi90/triviaqa) [![Dataset](https://img.shields.io/badge/Dataset-HuggingFace-ff9d00?logo=huggingface&logoColor=white)](https://huggingface.co/datasets/mandarjoshi/trivia_qa) > **Quick answer:** TriviaQA is a large-scale open-domain reading-comprehension benchmark built from roughly 95K real trivia questions paired with over 650K question-answer-evidence triples, created by Joshi et al. (2017). The official CodaLab leaderboard is saturated — the top system already reaches 94–96% F1 — making TriviaQA far less discriminative for modern frontier models than newer factual-QA benchmarks. --- ## At a Glance **What it tests:** Open-domain reading comprehension — finding and extracting the correct answer from noisy retrieved evidence documents for real trivia questions. **Why it matters:** It was one of the first QA benchmarks to pair questions with independently-gathered evidence rather than hand-written passages, making it harder and more realistic than earlier reading-comprehension sets. **Known limitations:** The official leaderboard is dominated by pre-LLM systems (2018–2020 era) and is effectively saturated; modern LLM scores are only available from third-party aggregators using a different, normalized metric. ## What TriviaQA Measures TriviaQA pairs real trivia questions with evidence documents gathered independently from the web and Wikipedia — meaning the evidence is noisier and less curated than hand-written reading-comprehension datasets. Systems must locate the answer span within this evidence, often requiring paraphrase handling and reasoning across multiple sentences rather than simple keyword matching. ## Benchmark Specifications | Field | Value | |---|---| | Question-answer pairs | ~95,000 | | Question-answer-evidence triples | 650,000+ | | Task category | Open-domain QA / reading comprehension | | Metric | Exact match (EM) and F1 — `full-em`, `full-f1`, `verified-em`, `verified-f1` | | Released | May 2017 | | Created by | Mandar Joshi, Eunsol Choi, Daniel S. Weld, Luke Zettlemoyer (University of Washington NLP) | ## How TriviaQA Is Scored The official metric is exact match (EM) and F1 between the predicted answer span and the ground-truth answer, computed over both the full dev/test set and a "verified" human-checked subset. The official [CodaLab leaderboard](https://competitions.codalab.org/competitions/17208#results) reports all four columns per submission. ## TriviaQA on Benchgen > No Benchgen results yet — [be the first to run TriviaQA](/benchmarks/triviaqa). ## TriviaQA vs Other Benchmarks | Benchmark | What it tests | Saturation | |---|---|---| | TriviaQA | Open-domain reading comprehension from trivia + evidence | High | | [Natural Questions](/benchmarks/natural-questions) | Open-domain QA from real Google search queries | Medium | | [DROP](/benchmarks/drop) | Discrete reasoning over paragraphs | — | ## Run TriviaQA on Your Model Benchgen lets you run TriviaQA against your own model with version-controlled, regression-tracked results, useful as a quick sanity check even though the benchmark is largely saturated for frontier models. ## Related Models & Benchmarks - [Natural Questions](/benchmarks/natural-questions) - [DROP](/benchmarks/drop) ## Frequently Asked Questions

What is TriviaQA?

TriviaQA is a reading-comprehension benchmark built from ~95K real trivia questions paired with independently-gathered evidence documents, created by Joshi et al. (2017).

What does a good score look like?

On the official EM/F1 leaderboard, top systems already reach 89–96%, so the benchmark is effectively saturated; it's no longer very discriminative for frontier LLMs.

Who created TriviaQA?

TriviaQA was created by Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer at the University of Washington, published in May 2017.

Rank	Model	Score
1	kimi-k2-base	0.851
2	gemma-2-27b	0.837
3	mimo-v2-5-pro	0.813

TriviaQA — Results

TriviaQA — Evaluations