DROP

DROP

Quick answer: DROP (Discrete Reasoning Over Paragraphs) is a reading comprehension benchmark from the Allen Institute for AI (Dua et al., 2019). It contains 96,567 question-answer pairs over Wikipedia passages that require discrete operations — counting, arithmetic, sorting, or set operations — to answer correctly. Unlike standard reading comprehension, DROP questions cannot be answered by finding a span in the text; the model must perform explicit numerical or logical operations. DROP is evaluated with F1 score and exact match.

At a Glance

What it tests: The ability to extract numerical or set-based information from text and perform discrete operations: counting entities, computing differences, sorting dates, finding intersections, and arithmetic over extracted values.

Why it matters: DROP separates models that merely find spans in text from models that can reason over extracted information. It was a major advance in reading comprehension difficulty when released in 2019 and remains a key benchmark for numerical reasoning over unstructured text.

Known limitations: DROP is now largely saturated for frontier models (90%+ F1). The benchmark tests specific types of discrete reasoning that don't cover open-ended or multi-hop reasoning well. Newer benchmarks like QuALITY and SCROLLS address harder long-context reasoning.

What DROP Measures

DROP uses Wikipedia passages as context, primarily from sports reports and history articles (chosen for their numerical density). Questions require one or more of these discrete operations:

• Counting: "How many players scored more than 10 points?"
• Arithmetic: "How many yards longer was the second field goal than the first?"
• Comparison/Sorting: "Who scored first, Smith or Jones?"
• Set operations: "Which players scored touchdowns but not field goals?"
• Date arithmetic: "How many years between Event A and Event B?"

Evaluation uses token-level F1 score and exact match. F1 gives partial credit for partially correct answers.

Benchmark Specifications

Frequently Asked Questions

What is DROP used to evaluate? DROP tests numerical and discrete reasoning grounded in natural language — can a model extract numbers from text and perform basic arithmetic or counting? It was designed to go beyond span extraction to require genuine symbolic manipulation.

What score do frontier models get on DROP? Frontier models (GPT-4o, Claude 3.5+, Gemini 1.5+) score above 90% F1 on DROP. Early large language models in 2019–2020 scored in the 40–60% range. The benchmark is now largely saturated.

How is DROP scored? DROP uses F1 score computed at the token level (treating the answer as a bag of tokens) and exact match. For numerical answers, an answer is correct if it produces the right number (regardless of formatting). F1 gives partial credit for partially correct multi-part answers.