PopQA
1 phaseActive
14K-question open-domain QA set testing long-tail entity knowledge; models struggle on low-popularity facts even at scale, motivating retrieval augmentation.
Overview
PopQA
Quick answer: PopQA is an open-domain question-answering dataset of 14,000 entity-centric questions spanning a wide range of real-world entity popularity, created to show that LLMs reliably know facts about popular entities but struggle badly on long-tail, low-popularity knowledge — a gap that scaling model size alone does not close.
At a Glance
What it tests: Whether a model's parametric memory reliably encodes factual knowledge about long-tail entities, not just the popular head of the distribution that dominates pretraining data and most benchmarks.
Why it matters: PopQA's central finding — that retrieval-augmented LMs substantially outperform much larger unassisted LMs specifically on long-tail knowledge, while unassisted LMs stay competitive on high-popularity entities — is one of the clearest empirical arguments for selectively applying retrieval augmentation rather than always or never using it.
Known limitations: Popularity is estimated with heuristics per entity type rather than a single universal measure, and the dataset's age (2022) means very recent, previously-uncommon entities aren't represented.
What PopQA Measures
PopQA questions are generated from knowledge-graph triples (subject entity, relation, object entity) filled into 16 relation-specific templates, covering a deliberately wide range of entity popularity — from globally famous entities to obscure, rarely-discussed ones. The dataset lets researchers directly measure how a model's factual accuracy degrades as entity popularity drops, holding question format constant.
The paper's key empirical result: LLMs of all scales struggle with long-tail knowledge — scaling model parameters fails to meaningfully close this gap — while retrieval-augmented models (even much smaller ones) substantially outperform larger unassisted LMs specifically on the low-popularity tail, while unassisted LMs remain competitive for high-popularity entities. This motivated the paper's proposed "adaptive retrieval" method: retrieving non-parametric context only when necessary (i.e., for lower-popularity entities), improving both performance and inference efficiency.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Reasoning / open-domain factual QA |
| Metric | Accuracy (exact match against gold entity) |
| Number of tasks | 14,000 questions across 16 relation types |
| Saturation | Medium — high-popularity subset is largely solved; long-tail subset is not |
| Created by | Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, Hannaneh Hajishirzi (University of Washington / Allen Institute for AI) |
| Source paper | Mallen et al. 2023 (ACL 2023) |
| GitHub | AlexTMallen/adaptive-retrieval |
| Dataset | akariasai/PopQA |
How PopQA Is Scored
Accuracy is measured as exact string match between the model's predicted answer and any of the gold aliases for the correct entity. Because PopQA is explicitly designed with a popularity axis, results are often reported broken down by popularity quartile/decile rather than as a single aggregate number, since aggregate accuracy can obscure a model's true long-tail weakness.
State-of-the-Art Results
Scores vary meaningfully by popularity bucket and retrieval configuration. See the official GitHub repository for the paper's full per-model, per-popularity-bucket breakdown.
PopQA on Benchgen
No Benchgen results yet — be the first to run PopQA.
PopQA vs Other Benchmarks
| Benchmark | What it tests | Tasks | Saturation |
|---|---|---|---|
| PopQA | Long-tail entity factual recall | 14,000 | medium |
| Natural Questions | Real Google search-query QA | — | medium |
| TriviaQA | Trivia-style factual QA | — | high |
PopQA is distinguished from Natural Questions and TriviaQA by deliberately engineering a wide, measurable spread of entity popularity, isolating the long-tail knowledge gap that aggregate accuracy on other QA sets can hide.
Run PopQA on Your Model
Benchgen lets teams evaluate their own model's factual accuracy broken down by entity popularity, surfacing long-tail knowledge gaps that a single aggregate QA score would mask — and testing whether retrieval augmentation actually closes that gap.