# AI2 Reasoning Challenge (ARC) ![Category](https://img.shields.io/badge/Category-Science%20Reasoning-1f6feb) ![Metric](https://img.shields.io/badge/Metric-Accuracy-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-7%2C787-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-High-d29922) ![Created](https://img.shields.io/badge/Created-Mar%202018-555) [![Paper](https://img.shields.io/badge/Paper-arXiv%201803.05457-b31b1b)](https://arxiv.org/abs/1803.05457) [![GitHub](https://img.shields.io/badge/GitHub-allenai%2FARC--Solvers-181717?logo=github&logoColor=white)](https://github.com/allenai/ARC-Solvers) [![Dataset](https://img.shields.io/badge/Dataset-HuggingFace-ff9d00?logo=huggingface&logoColor=white)](https://huggingface.co/datasets/allenai/ai2_arc) > **Quick answer:** The AI2 Reasoning Challenge (ARC), introduced by Clark et al. at the Allen Institute for AI in March 2018, is a benchmark of 7,787 genuine grade-school science multiple-choice questions partitioned into an Easy Set and a Challenge Set — the Challenge Set contains only questions that retrieval-based systems and word co-occurrence algorithms both failed to answer, making it a test of genuine scientific reasoning rather than surface-level pattern matching. ## At a Glance **What it tests:** Multi-choice grade-school science reasoning across biology, physics, earth science, and chemistry, requiring causal understanding and conceptual knowledge that cannot be answered by simple keyword matching. **Why it matters:** ARC was one of the first benchmarks to specifically filter out questions solvable by shallow retrieval, establishing a principled difficulty floor. It helped define what "reasoning" means for LLMs and served as a standard evaluation fixture for multiple generations of models. **Known limitations:** The benchmark is now heavily saturated — GPT-4 achieves 96.3% on the Challenge Set, and most current frontier models score above 90%. It no longer meaningfully differentiates state-of-the-art models. ## What ARC Measures ARC consists of 7,787 genuine questions sourced from U.S. grade 3–9 science exams. The questions span four major scientific domains: biology, physics, earth science, and chemistry, with coverage of conceptual knowledge, causal reasoning, and experimental design. Each question is a four-way multiple-choice item. The benchmark is divided into two partitions. The **Easy Set** contains questions that at least one of the baseline retrieval systems answered correctly. The **Challenge Set** contains 2,590 questions that no retrieval-based system and no word co-occurrence algorithm answered correctly at the time of publication — ensuring these questions require genuine reasoning. The Challenge Set is the standard subset reported in most leaderboard comparisons, often referred to as "ARC-C." A supporting corpus of over 14 million science-related sentences accompanies the dataset, originally designed to support retrieval-augmented approaches but now primarily relevant as historical context. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Science reasoning | | Metric | Accuracy | | Total questions | 7,787 | | Challenge Set (ARC-C) | 2,590 | | Easy Set (ARC-E) | 5,197 | | Saturation | High | | Created by | Clark et al. (Allen Institute for AI) | | Source paper | [Clark et al. 2018](https://arxiv.org/abs/1803.05457) | | GitHub | [allenai/ARC-Solvers](https://github.com/allenai/ARC-Solvers) | | Dataset | [allenai/ai2_arc](https://huggingface.co/datasets/allenai/ai2_arc) | ## How ARC Is Scored Each question is scored correct or incorrect based on the model's multiple-choice answer selection. The overall accuracy is the fraction of correct answers across the question set. Results are typically reported separately for ARC-C (Challenge) and ARC-E (Easy), with ARC-C being the standard figure cited in model cards and comparisons. ## ARC on Benchgen > No Benchgen results yet — [be the first to run ARC](/benchmarks/allenai/ai2-reasoning-challenge). ## ARC vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **ARC (Challenge Set)** | Grade-school science reasoning | 2,590 | High | | [GPQA Diamond](/benchmarks/rein-et-al/gpqa-diamond) | Expert-level science reasoning | 198 | Low | | [MMLU Pro](/benchmarks/tiger-ai-lab-university-of-waterloo/mmlu-pro) | Multi-domain professional knowledge | 12,000 | Medium | | [Humanity's Last Exam](/benchmarks/center-for-ai-safety-scale-ai/humanitys-last-exam) | Expert-level multidisciplinary knowledge | 3,000 | Low | ARC's grade-school difficulty means it is effectively saturated for frontier models. For meaningful differentiation of current-generation models, use GPQA Diamond or Humanity's Last Exam instead. ## Run ARC on Your Model While ARC is saturated for frontier models, it remains a useful regression check for smaller or fine-tuned models. Benchgen lets you run ARC as part of a baseline evaluation suite and track results across model versions to verify that fine-tuning hasn't degraded basic reasoning capability. ## Frequently Asked Questions

What is the ARC benchmark?

The AI2 Reasoning Challenge (ARC) is a dataset of 7,787 grade-school science multiple-choice questions created by Clark et al. at the Allen Institute for AI in 2018. The Challenge Set (ARC-C) of 2,590 questions is specifically filtered to exclude questions solvable by retrieval, requiring genuine scientific reasoning.

What does a good score look like on ARC?

ARC is now heavily saturated. GPT-4 achieves 96.3% on the Challenge Set, and most frontier models score above 90%. Scores below 85% indicate meaningful reasoning limitations; scores above 95% offer no differentiation between frontier models.

Who created ARC?

ARC was created by Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, and colleagues at the Allen Institute for AI (AllenAI). The paper was published in March 2018 (arXiv:1803.05457).

Rank	Model	Score
1	deepseek-r1-0528	0.981
2	qwen2-5-72b-instruct	0.979
3	gpt-4-1	0.971
4	llama-3-1-405b-instruct	0.969
5	claude-3-5-sonnet	0.967
6	claude-3-opus	0.964
7	r1	0.964
8	gpt-4-0613	0.963
9	gpt-4o	0.963
10	phi-4	0.96
11	llama-3-3-70b-instruct	0.96
12	o4-mini	0.959
13	llama-3-1-70b-instruct	0.948
14	llama-4-maverick	0.938
15	gemini-1-5-pro	0.914
16	llama-4-scout	0.907

AI2 Reasoning Challenge (ARC) — Results

AI2 Reasoning Challenge (ARC) — Evaluations