TriviaQA — Results
| Rank | Model | Score |
|---|---|---|
| 1 | kimi-k2-base | 0.851 |
| 2 | gemma-2-27b | 0.837 |
| 3 | mimo-v2-5-pro | 0.813 |
TriviaQA
1 phaseActive
Large-scale open-domain reading-comprehension benchmark built from trivia questions plus retrieved evidence. Metric: exact match / F1. Official leaderboard is saturated.
Overview
TriviaQA
Quick answer: TriviaQA is a large-scale open-domain reading-comprehension benchmark built from roughly 95K real trivia questions paired with over 650K question-answer-evidence triples, created by Joshi et al. (2017). The official CodaLab leaderboard is saturated — the top system already reaches 94–96% F1 — making TriviaQA far less discriminative for modern frontier models than newer factual-QA benchmarks.
At a Glance
What it tests: Open-domain reading comprehension — finding and extracting the correct answer from noisy retrieved evidence documents for real trivia questions. Why it matters: It was one of the first QA benchmarks to pair questions with independently-gathered evidence rather than hand-written passages, making it harder and more realistic than earlier reading-comprehension sets. Known limitations: The official leaderboard is dominated by pre-LLM systems (2018–2020 era) and is effectively saturated; modern LLM scores are only available from third-party aggregators using a different, normalized metric.
What TriviaQA Measures
TriviaQA pairs real trivia questions with evidence documents gathered independently from the web and Wikipedia — meaning the evidence is noisier and less curated than hand-written reading-comprehension datasets. Systems must locate the answer span within this evidence, often requiring paraphrase handling and reasoning across multiple sentences rather than simple keyword matching.
Benchmark Specifications
| Field | Value |
|---|---|
| Question-answer pairs | ~95,000 |
| Question-answer-evidence triples | 650,000+ |
| Task category | Open-domain QA / reading comprehension |
| Metric | Exact match (EM) and F1 — full-em, full-f1, verified-em, verified-f1 |
| Released | May 2017 |
| Created by | Mandar Joshi, Eunsol Choi, Daniel S. Weld, Luke Zettlemoyer (University of Washington NLP) |
How TriviaQA Is Scored
The official metric is exact match (EM) and F1 between the predicted answer span and the ground-truth answer, computed over both the full dev/test set and a "verified" human-checked subset. The official CodaLab leaderboard reports all four columns per submission.
TriviaQA on Benchgen
No Benchgen results yet — be the first to run TriviaQA.
TriviaQA vs Other Benchmarks
| Benchmark | What it tests | Saturation |
|---|---|---|
| TriviaQA | Open-domain reading comprehension from trivia + evidence | High |
| Natural Questions | Open-domain QA from real Google search queries | Medium |
| DROP | Discrete reasoning over paragraphs | — |
Run TriviaQA on Your Model
Benchgen lets you run TriviaQA against your own model with version-controlled, regression-tracked results, useful as a quick sanity check even though the benchmark is largely saturated for frontier models.