OpenBookQA
1 phaseActive
Elementary-science open-book QA dataset (~6,000 questions over 1,329 facts) requiring common-knowledge combination, created by AI2 (EMNLP 2018).
Overview
OpenBookQA
Quick answer: OpenBookQA is a question-answering dataset modeled after open-book exams, pairing roughly 6,000 elementary-science questions with a fixed "open book" of 1,329 core science facts, requiring models to combine a given fact with broad common knowledge to answer correctly.
At a Glance
What it tests: Multi-hop reasoning that combines an explicit fact (e.g., "metals conduct electricity") with unstated common knowledge (e.g., "a suit of armor is made of metal") to reach a conclusion neither fact alone supports.
Why it matters: Unlike closed-book factual recall, OpenBookQA specifically probes whether a model can apply a given fact to a genuinely novel situation — closer to real reasoning than memorization.
Known limitations: Published in 2018, the dataset is long since saturated by frontier LLMs and carries meaningful contamination risk given its age and public availability.
What OpenBookQA Measures
Each OpenBookQA question comes with an associated fact from the 1,329-fact "book," but answering correctly requires combining that fact with additional common knowledge not stated anywhere in the question or book. The task was explicitly designed to go beyond datasets that can be solved by simple retrieval or lexical matching over a passage — models need genuine common-sense reasoning layered on top of the given scientific fact.
Human performance on the original test set is close to 92%, but the original 2018 paper found several pretrained QA baselines performed surprisingly poorly, worse than simple neural baselines built specifically for the task — evidence that surface-level retrieval methods struggle with the multi-hop combination step even when the primary fact is provided.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Reasoning / open-domain science QA |
| Metric | Accuracy |
| Number of tasks | 5,957 questions (~6,000) over 1,329 core facts |
| Saturation | High — long-saturated by frontier LLMs, contamination risk |
| Created by | Todor Mihaylov, Peter Clark, Tushar Khot, Ashish Sabharwal (Allen Institute for AI) |
| Source paper | Mihaylov et al. 2018 |
| Dataset | allenai/openbookqa |
How OpenBookQA Is Scored
Each question is multiple-choice (4 options); accuracy is the fraction of questions answered correctly. Human performance sits near 92%, which serves as the informal ceiling most model comparisons reference.
State-of-the-Art Results
| Rank | Model | Score | Source | Date |
|---|---|---|---|---|
| — | Human performance | ~92% | Mihaylov et al. 2018 | 2018-09 |
Modern frontier LLMs are widely reported to exceed 95% accuracy on OpenBookQA in third-party aggregations; Benchgen has not yet independently verified per-model current-generation scores for this legacy 2018 dataset — see OpenBookQA on Benchgen to be the first to submit a verified run.
OpenBookQA on Benchgen
No Benchgen results yet — be the first to run OpenBookQA.
OpenBookQA vs Other Benchmarks
| Benchmark | What it tests | Tasks | Saturation |
|---|---|---|---|
| OpenBookQA | Fact + common-knowledge multi-hop QA | 5,957 | high |
| ARC-C | Grade-school science QA (harder split) | — | high |
| PopQA | Open-domain entity factual recall | 14,000 | medium |
OpenBookQA sits alongside ARC as one of the classic AI2 science-QA benchmarks, distinguished by explicitly providing the relevant fact and testing the common-knowledge combination step rather than fact retrieval itself.
Run OpenBookQA on Your Model
Benchgen lets teams verify OpenBookQA accuracy for their own model versions on demand, providing an apples-to-apples comparison against the historical human baseline and other models rather than relying on scattered, inconsistently-reported third-party numbers.