ARC-E — Results
| Rank | Model | Score |
|---|---|---|
| 1 | gemma-2-27b | 0.886 |
| 2 | gemma-2-9b | 0.88 |
| 3 | hermes-3-70b | 0.83 |
| 4 | gemma-3n-e4b-litert-preview | 0.816 |
| 5 | gemma-3n-e4b | 0.816 |
| 6 | gemma-3n-e2b-litert-preview | 0.758 |
| 7 | gemma-3n-e2b | 0.758 |
| 8 | ernie-4-5 | 0.607 |
ARC-E
1 phaseActive
AI2 Reasoning Challenge easy subset: 2376 science MCQs answerable by retrieval methods. Accuracy metric, high saturation. Clark et al. 2018.
Overview
ARC-E
Quick answer: ARC-E (AI2 Reasoning Challenge — Easy Set) is the accessible half of the AI2 Reasoning Challenge, containing 2376 grade-school science multiple-choice questions that at least one retrieval-based method could answer. It measures basic scientific knowledge and factual reasoning. Introduced by Clark et al. in 2018, it is highly saturated at the frontier with only 8 models evaluated — the leader Gemma 2 27B scores 88.6%.
At a Glance
What it tests: Basic scientific knowledge and factual recall through grade-school level multiple-choice questions. Unlike ARC-C, these questions are simpler and can be answered by retrieval-based systems.
Why it matters: ARC-E provides a baseline scientific literacy test for smaller models and serves as a complementary lower bound when used alongside ARC-C. It is especially useful for establishing that a model has absorbed basic scientific knowledge during pre-training.
Known limitations: Highly saturated — only 8 models have been evaluated and the benchmark no longer differentiates frontier capabilities meaningfully. ARC-C or GPQA Diamond are better choices for frontier evaluation.
What ARC-E Measures
ARC-E is the Easy Set partition of the AI2 Reasoning Challenge dataset. It contains questions where at least one of a retrieval-based system or a word co-occurrence system answered correctly. This makes ARC-E substantially more accessible than ARC-C — the questions can often be solved by knowing the right keywords rather than deriving an answer through multi-step reasoning.
The 2376-question test split covers the same domains as ARC-C: life science, Earth science, physical science, and general scientific knowledge at a grade 3–9 level. The multiple-choice format (3–4 options) is identical to ARC-C.
ARC-E is primarily useful for evaluating smaller language models, quantised variants, or edge-deployed models where basic scientific knowledge retention is the relevant signal.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Reasoning (scientific knowledge) |
| Metric | Accuracy (fraction correct, 0–1) |
| Number of tasks | 2376 (test split), 5197 (full split) |
| Format | Multiple-choice (3–4 options) |
| Domain | Grade-school science (grades 3–9) |
| Saturation | High |
| Created by | Clark et al. (Allen Institute for AI) |
| Source paper | Clark et al. 2018 |
| Dataset | allenai/ai2_arc |
How ARC-E Is Scored
Each question is scored as correct (1) or incorrect (0) based on the selected multiple-choice answer. The final score is the fraction of questions answered correctly. No retrieval or external tools are permitted. Scores range from 0–1 (reported as percentages in some leaderboards).
ARC-E on Benchgen
No Benchgen results yet — be the first to run ARC-E.
ARC-E vs Other Benchmarks
| Benchmark | What it tests | Tasks | Saturation |
|---|---|---|---|
| ARC-E | Grade-school science (easy) | 2376 | High |
| ARC-C | Grade-school science (challenge) | 1119 | High |
| MMLU | Multidomain knowledge (57 subjects) | 14042 | High |
| GPQA Diamond | Expert-level science | 198 | Low |
Use ARC-E alongside ARC-C to evaluate a model's full spectrum of basic scientific knowledge. For frontier evaluation, GPQA Diamond or MMLU-Pro provide better signal.
Run ARC-E on Your Model
Benchgen makes it straightforward to include ARC-E as a regression checkpoint in your evaluation pipeline — confirming that fine-tuning or quantisation has not degraded basic scientific knowledge retrieval.
Related Models & Benchmarks
- Related benchmarks: ARC-C · MMLU · MMLU-Pro · GPQA Diamond
Frequently Asked Questions
What is the ARC-E benchmark?
ARC-E (AI2 Reasoning Challenge — Easy Set) is the accessible subset of the ARC dataset, containing 2376 grade-school science multiple-choice questions that at least one retrieval-based system could answer. It tests basic scientific knowledge and factual recall. See Clark et al. 2018.What does a good ARC-E score look like?
Capable models typically score 0.75–0.89. The current leader scores 0.886. ARC-E is less discriminating than ARC-C for frontier models — most capable models perform similarly.Who created ARC-E?
ARC-E shares the same authors as ARC-C: Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, and colleagues at the Allen Institute for AI. Published March 2018 — see arXiv 1803.05457.Is ARC-E saturated?
Yes. Very few models have been evaluated on ARC-E (8 as of mid-2026) and the benchmark no longer differentiates frontier capabilities. It is primarily useful for smaller models or basic regression testing.How does ARC-E differ from ARC-C?
ARC-E contains questions that retrieval-based methods could answer — making them generally easier. ARC-C filters for questions that defeated retrieval methods, requiring genuine reasoning. Both benchmarks share the same paper, dataset, and multiple-choice format.Benchmark definition paraphrased from Clark et al. 2018. Last updated 2026-07-23.