# PopQA ![Category](https://img.shields.io/badge/Category-Reasoning-1f6feb) ![Metric](https://img.shields.io/badge/Metric-Accuracy-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-14%2C000-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-medium-d29922) ![Created](https://img.shields.io/badge/Created-Dec%202022-555) [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/abs/2212.10511) [![GitHub](https://img.shields.io/badge/GitHub-AlexTMallen%2Fadaptive--retrieval-181717?logo=github&logoColor=white)](https://github.com/AlexTMallen/adaptive-retrieval) [![Dataset](https://img.shields.io/badge/Dataset-HuggingFace-ff9d00?logo=huggingface&logoColor=white)](https://huggingface.co/datasets/akariasai/PopQA) > **Quick answer:** PopQA is an open-domain question-answering dataset of 14,000 entity-centric questions spanning a wide range of real-world entity popularity, created to show that LLMs reliably know facts about popular entities but struggle badly on long-tail, low-popularity knowledge — a gap that scaling model size alone does not close. ## At a Glance **What it tests:** Whether a model's parametric memory reliably encodes factual knowledge about long-tail entities, not just the popular head of the distribution that dominates pretraining data and most benchmarks. **Why it matters:** PopQA's central finding — that retrieval-augmented LMs substantially outperform much larger unassisted LMs specifically on long-tail knowledge, while unassisted LMs stay competitive on high-popularity entities — is one of the clearest empirical arguments for selectively applying retrieval augmentation rather than always or never using it. **Known limitations:** Popularity is estimated with heuristics per entity type rather than a single universal measure, and the dataset's age (2022) means very recent, previously-uncommon entities aren't represented. ## What PopQA Measures PopQA questions are generated from knowledge-graph triples (subject entity, relation, object entity) filled into 16 relation-specific templates, covering a deliberately wide range of entity popularity — from globally famous entities to obscure, rarely-discussed ones. The dataset lets researchers directly measure how a model's factual accuracy degrades as entity popularity drops, holding question format constant. The paper's key empirical result: LLMs of all scales struggle with long-tail knowledge — scaling model parameters fails to meaningfully close this gap — while retrieval-augmented models (even much smaller ones) substantially outperform larger unassisted LMs specifically on the low-popularity tail, while unassisted LMs remain competitive for high-popularity entities. This motivated the paper's proposed "adaptive retrieval" method: retrieving non-parametric context only when necessary (i.e., for lower-popularity entities), improving both performance and inference efficiency. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Reasoning / open-domain factual QA | | Metric | Accuracy (exact match against gold entity) | | Number of tasks | 14,000 questions across 16 relation types | | Saturation | Medium — high-popularity subset is largely solved; long-tail subset is not | | Created by | Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, Hannaneh Hajishirzi (University of Washington / Allen Institute for AI) | | Source paper | [Mallen et al. 2023](https://arxiv.org/abs/2212.10511) (ACL 2023) | | GitHub | [AlexTMallen/adaptive-retrieval](https://github.com/AlexTMallen/adaptive-retrieval) | | Dataset | [akariasai/PopQA](https://huggingface.co/datasets/akariasai/PopQA) | ## How PopQA Is Scored Accuracy is measured as exact string match between the model's predicted answer and any of the gold aliases for the correct entity. Because PopQA is explicitly designed with a popularity axis, results are often reported broken down by popularity quartile/decile rather than as a single aggregate number, since aggregate accuracy can obscure a model's true long-tail weakness. ## State-of-the-Art Results > Scores vary meaningfully by popularity bucket and retrieval configuration. See the [official GitHub repository](https://github.com/AlexTMallen/adaptive-retrieval) for the paper's full per-model, per-popularity-bucket breakdown. ## PopQA on Benchgen > No Benchgen results yet — [be the first to run PopQA](/benchmarks/alex-mallen-et-al-university-of-washington-allen-institute-for-ai/popqa). ## PopQA vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **PopQA** | Long-tail entity factual recall | 14,000 | medium | | [Natural Questions](/benchmarks/natural-questions) | Real Google search-query QA | — | medium | | [TriviaQA](/benchmarks/triviaqa) | Trivia-style factual QA | — | high | PopQA is distinguished from Natural Questions and TriviaQA by deliberately engineering a wide, measurable spread of entity popularity, isolating the long-tail knowledge gap that aggregate accuracy on other QA sets can hide. ## Run PopQA on Your Model Benchgen lets teams evaluate their own model's factual accuracy broken down by entity popularity, surfacing long-tail knowledge gaps that a single aggregate QA score would mask — and testing whether retrieval augmentation actually closes that gap. ## Frequently Asked Questions

What is PopQA?

PopQA is a 14,000-question open-domain QA dataset that deliberately spans a wide range of entity popularity, used to measure how model factual accuracy degrades for long-tail, low-popularity entities.

What does a good PopQA score look like?

Most models score well on high-popularity entities but drop sharply on long-tail entities; a genuinely strong result means maintaining high accuracy across the full popularity spectrum, not just the aggregate average.

Who created PopQA?

PopQA was created by Alex Mallen and collaborators at the University of Washington and Allen Institute for AI, published at ACL 2023.

PopQA — Phases

PopQA