AI2 Reasoning Challenge (ARC)

AI2 Reasoning Challenge (ARC)

Quick answer: The AI2 Reasoning Challenge (ARC), introduced by Clark et al. at the Allen Institute for AI in March 2018, is a benchmark of 7,787 genuine grade-school science multiple-choice questions partitioned into an Easy Set and a Challenge Set — the Challenge Set contains only questions that retrieval-based systems and word co-occurrence algorithms both failed to answer, making it a test of genuine scientific reasoning rather than surface-level pattern matching.

At a Glance

What it tests: Multi-choice grade-school science reasoning across biology, physics, earth science, and chemistry, requiring causal understanding and conceptual knowledge that cannot be answered by simple keyword matching.

Why it matters: ARC was one of the first benchmarks to specifically filter out questions solvable by shallow retrieval, establishing a principled difficulty floor. It helped define what "reasoning" means for LLMs and served as a standard evaluation fixture for multiple generations of models.

Known limitations: The benchmark is now heavily saturated — GPT-4 achieves 96.3% on the Challenge Set, and most current frontier models score above 90%. It no longer meaningfully differentiates state-of-the-art models.

What ARC Measures

ARC consists of 7,787 genuine questions sourced from U.S. grade 3–9 science exams. The questions span four major scientific domains: biology, physics, earth science, and chemistry, with coverage of conceptual knowledge, causal reasoning, and experimental design. Each question is a four-way multiple-choice item.

The benchmark is divided into two partitions. The Easy Set contains questions that at least one of the baseline retrieval systems answered correctly. The Challenge Set contains 2,590 questions that no retrieval-based system and no word co-occurrence algorithm answered correctly at the time of publication — ensuring these questions require genuine reasoning. The Challenge Set is the standard subset reported in most leaderboard comparisons, often referred to as "ARC-C."

A supporting corpus of over 14 million science-related sentences accompanies the dataset, originally designed to support retrieval-augmented approaches but now primarily relevant as historical context.

Benchmark Specifications

How ARC Is Scored

Each question is scored correct or incorrect based on the model's multiple-choice answer selection. The overall accuracy is the fraction of correct answers across the question set. Results are typically reported separately for ARC-C (Challenge) and ARC-E (Easy), with ARC-C being the standard figure cited in model cards and comparisons.

ARC on Benchgen

No Benchgen results yet — be the first to run ARC.

ARC vs Other Benchmarks

ARC's grade-school difficulty means it is effectively saturated for frontier models. For meaningful differentiation of current-generation models, use GPQA Diamond or Humanity's Last Exam instead.

Run ARC on Your Model

While ARC is saturated for frontier models, it remains a useful regression check for smaller or fine-tuned models. Benchgen lets you run ARC as part of a baseline evaluation suite and track results across model versions to verify that fine-tuning hasn't degraded basic reasoning capability.

Frequently Asked Questions