# ARC-E ![Category](https://img.shields.io/badge/Category-Reasoning-1f6feb) ![Metric](https://img.shields.io/badge/Metric-Accuracy-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-2376-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-High-d29922) ![Created](https://img.shields.io/badge/Created-Mar%202018-555) [![Paper](https://img.shields.io/badge/Paper-arXiv%201803.05457-b31b1b)](https://arxiv.org/abs/1803.05457) [![Dataset](https://img.shields.io/badge/Dataset-HuggingFace-ff9d00?logo=huggingface&logoColor=white)](https://huggingface.co/datasets/allenai/ai2_arc) > **Quick answer:** ARC-E (AI2 Reasoning Challenge — Easy Set) is the accessible half of the AI2 Reasoning Challenge, containing 2376 grade-school science multiple-choice questions that at least one retrieval-based method could answer. It measures basic scientific knowledge and factual reasoning. Introduced by Clark et al. in 2018, it is highly saturated at the frontier with only 8 models evaluated — the leader Gemma 2 27B scores 88.6%. ## At a Glance **What it tests:** Basic scientific knowledge and factual recall through grade-school level multiple-choice questions. Unlike ARC-C, these questions are simpler and can be answered by retrieval-based systems. **Why it matters:** ARC-E provides a baseline scientific literacy test for smaller models and serves as a complementary lower bound when used alongside ARC-C. It is especially useful for establishing that a model has absorbed basic scientific knowledge during pre-training. **Known limitations:** Highly saturated — only 8 models have been evaluated and the benchmark no longer differentiates frontier capabilities meaningfully. ARC-C or GPQA Diamond are better choices for frontier evaluation. ## What ARC-E Measures ARC-E is the Easy Set partition of the AI2 Reasoning Challenge dataset. It contains questions where at least one of a retrieval-based system or a word co-occurrence system answered correctly. This makes ARC-E substantially more accessible than ARC-C — the questions can often be solved by knowing the right keywords rather than deriving an answer through multi-step reasoning. The 2376-question test split covers the same domains as ARC-C: life science, Earth science, physical science, and general scientific knowledge at a grade 3–9 level. The multiple-choice format (3–4 options) is identical to ARC-C. ARC-E is primarily useful for evaluating smaller language models, quantised variants, or edge-deployed models where basic scientific knowledge retention is the relevant signal. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Reasoning (scientific knowledge) | | Metric | Accuracy (fraction correct, 0–1) | | Number of tasks | 2376 (test split), 5197 (full split) | | Format | Multiple-choice (3–4 options) | | Domain | Grade-school science (grades 3–9) | | Saturation | High | | Created by | Clark et al. (Allen Institute for AI) | | Source paper | [Clark et al. 2018](https://arxiv.org/abs/1803.05457) | | Dataset | [allenai/ai2_arc](https://huggingface.co/datasets/allenai/ai2_arc) | ## How ARC-E Is Scored Each question is scored as correct (1) or incorrect (0) based on the selected multiple-choice answer. The final score is the fraction of questions answered correctly. No retrieval or external tools are permitted. Scores range from 0–1 (reported as percentages in some leaderboards). ## ARC-E on Benchgen > No Benchgen results yet — [be the first to run ARC-E](/benchmarks/allenai/arc-e). ## ARC-E vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **ARC-E** | Grade-school science (easy) | 2376 | High | | [ARC-C](/benchmarks/clark-et-al-allen-institute-for-ai/arc-c) | Grade-school science (challenge) | 1119 | High | | [MMLU](/benchmarks/hendrycks-et-al/mmlu) | Multidomain knowledge (57 subjects) | 14042 | High | | [GPQA Diamond](/benchmarks/rein-et-al/gpqa-diamond) | Expert-level science | 198 | Low | Use ARC-E alongside ARC-C to evaluate a model's full spectrum of basic scientific knowledge. For frontier evaluation, GPQA Diamond or MMLU-Pro provide better signal. ## Run ARC-E on Your Model Benchgen makes it straightforward to include ARC-E as a regression checkpoint in your evaluation pipeline — confirming that fine-tuning or quantisation has not degraded basic scientific knowledge retrieval. ## Related Models & Benchmarks - Related benchmarks: [ARC-C](/benchmarks/clark-et-al-allen-institute-for-ai/arc-c) · [MMLU](/benchmarks/hendrycks-et-al/mmlu) · [MMLU-Pro](/benchmarks/tiger-ai-lab-university-of-waterloo/mmlu-pro) · [GPQA Diamond](/benchmarks/rein-et-al/gpqa-diamond) ## Frequently Asked Questions

What is the ARC-E benchmark?

ARC-E (AI2 Reasoning Challenge — Easy Set) is the accessible subset of the ARC dataset, containing 2376 grade-school science multiple-choice questions that at least one retrieval-based system could answer. It tests basic scientific knowledge and factual recall. See [Clark et al. 2018](https://arxiv.org/abs/1803.05457).

What does a good ARC-E score look like?

Capable models typically score 0.75–0.89. The current leader scores 0.886. ARC-E is less discriminating than ARC-C for frontier models — most capable models perform similarly.

Who created ARC-E?

ARC-E shares the same authors as ARC-C: Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, and colleagues at the Allen Institute for AI. Published March 2018 — see [arXiv 1803.05457](https://arxiv.org/abs/1803.05457).

Is ARC-E saturated?

Yes. Very few models have been evaluated on ARC-E (8 as of mid-2026) and the benchmark no longer differentiates frontier capabilities. It is primarily useful for smaller models or basic regression testing.

How does ARC-E differ from ARC-C?

ARC-E contains questions that retrieval-based methods could answer — making them generally easier. ARC-C filters for questions that defeated retrieval methods, requiring genuine reasoning. Both benchmarks share the same paper, dataset, and multiple-choice format.

--- *Benchmark definition paraphrased from [Clark et al. 2018](https://arxiv.org/abs/1803.05457). Last updated 2026-07-23.*

Rank	Model	Score
1	gemma-2-27b	0.886
2	gemma-2-9b	0.88
3	hermes-3-70b	0.83
4	gemma-3n-e4b-litert-preview	0.816
5	gemma-3n-e4b	0.816
6	gemma-3n-e2b-litert-preview	0.758
7	gemma-3n-e2b	0.758
8	ernie-4-5	0.607

ARC-E — Results

ARC-E — Evaluations