ARC-C — Results
| Rank | Model | Score |
|---|---|---|
| 1 | mimo-v2-5-pro | 0.972 |
| 2 | llama-3-1-405b-instruct | 0.969 |
| 3 | claude-3-opus | 0.964 |
| 4 | llama-3-1-70b-instruct | 0.948 |
| 5 | nova-2-pro | 0.948 |
| 6 | claude-3-sonnet | 0.932 |
| 7 | jamba-1-5-large | 0.93 |
| 8 | nova-2-lite | 0.924 |
| 9 | mistral-small-3-24b-base | 0.913 |
| 10 | phi-3-5-moe-instruct | 0.91 |
| 11 | claude-3-haiku | 0.892 |
| 12 | jamba-1-5-mini | 0.857 |
| 13 | phi-3-5-mini-instruct | 0.846 |
| 14 | phi-4-mini | 0.837 |
| 15 | llama-3-1-8b-instruct | 0.834 |
| 16 | qwen2-5-coder-32b-instruct | 0.705 |
| 17 | qwen2-5-32b-instruct | 0.704 |
| 18 | qwen2-72b-instruct | 0.689 |
| 19 | qwen2-5-14b-instruct | 0.673 |
| 20 | hermes-3-70b | 0.655 |
| 21 | gemma-3n-e4b-litert-preview | 0.616 |
| 22 | gemma-3n-e4b | 0.616 |
| 23 | qwen2-5-coder-7b-instruct | 0.609 |
| 24 | gemma-3n-e2b-litert-preview | 0.517 |
| 25 | gemma-3n-e2b | 0.517 |
ARC-C
1 phaseActive
AI2 Reasoning Challenge hard subset: 1119 science MCQs that stump retrieval-based methods. Accuracy metric, high saturation for frontier models. Clark et al. 2018.
Overview
ARC-C
Quick answer: ARC-C (AI2 Reasoning Challenge — Challenge Set) is a multiple-choice science reasoning benchmark containing 1119 grade-school questions specifically curated to defeat retrieval-based and word co-occurrence algorithms. Introduced by Clark et al. in 2018, it requires genuine commonsense and scientific reasoning. It is now highly saturated for frontier models — MiMo-V2.5-Pro leads at 97.2%.
At a Glance
What it tests: Commonsense and scientific reasoning through grade-school level multiple-choice questions that cannot be solved by simple keyword matching or document retrieval.
Why it matters: ARC-C's Challenge Set was specifically filtered to remove questions answerable by retrieval or word co-occurrence, making it a purer test of reasoning. It became a standard evaluation in open-source model comparisons and the Open LLM Leaderboard era, building topical authority for comparing mid-tier models.
Known limitations: Highly saturated — top frontier models exceed 96% accuracy. ARC-C no longer differentiates between capable models and is primarily useful for comparing smaller or older models.
What ARC-C Measures
The AI2 Reasoning Challenge (ARC) was designed to push beyond retrieval-based question answering toward genuine reasoning. The dataset draws from science question sources typically used in grades 3–9 and curates them into two subsets: the Easy Set (ARC-E) and the Challenge Set (ARC-C).
ARC-C contains 1119 test questions (2590 total in the full split) that both a retrieval-based system and a word co-occurrence system answered incorrectly — meaning they cannot be solved without actual reasoning. Questions cover topics in life science, Earth science, physical science, and general scientific knowledge.
The format is 3- or 4-option multiple choice. Models must select the correct answer from the provided choices, making partial solutions impossible and the metric simple: accuracy over all test questions.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Reasoning (commonsense + scientific) |
| Metric | Accuracy (fraction correct, 0–1) |
| Number of tasks | 1119 (test split), 2590 (full) |
| Format | Multiple-choice (3–4 options) |
| Domain | Grade-school science (grades 3–9) |
| Saturation | High |
| Created by | Clark et al. (Allen Institute for AI) |
| Source paper | Clark et al. 2018 |
| Dataset | allenai/ai2_arc |
How ARC-C Is Scored
Each question is scored as correct (1) or incorrect (0). The overall score is the fraction of questions answered correctly. Models are given the question and the multiple-choice options; no external retrieval is permitted. The evaluation is fully standardised and reproducible.
A score above 0.90 is typical for capable instruction-tuned models. Frontier models now score above 0.95, making ARC-C most useful for evaluating smaller models or comparing against older baselines.
ARC-C on Benchgen
No Benchgen results yet — be the first to run ARC-C.
ARC-C vs Other Benchmarks
| Benchmark | What it tests | Tasks | Saturation |
|---|---|---|---|
| ARC-C | Grade-school science reasoning (hard) | 1119 | High |
| ARC-E | Grade-school science reasoning (easy) | 2376 | High |
| MMLU | Multidomain knowledge (57 subjects) | 14042 | High |
| MMLU-Pro | Harder multidomain reasoning | 12032 | Medium |
| GPQA Diamond | Expert-level science | 198 | Low |
ARC-C is best used for tracking progress in smaller or fine-tuned models. For frontier model comparisons, MMLU-Pro or GPQA Diamond provide more differentiation.
Run ARC-C on Your Model
Use Benchgen to track ARC-C performance across fine-tuning iterations or prompt changes. While ARC-C is saturated for frontier models, it is a useful regression test to confirm that model updates have not degraded basic scientific reasoning capability.
Related Models & Benchmarks
- Top performers: MiMo-V2.5-Pro · Llama 3.1 405B Instruct · Claude 3 Opus
- Related benchmarks: ARC-E · MMLU · MMLU-Pro · GPQA Diamond
Frequently Asked Questions
What is the ARC-C benchmark?
ARC-C (AI2 Reasoning Challenge — Challenge Set) is a multiple-choice science reasoning benchmark containing 1119 grade-school questions that cannot be answered by retrieval-based or word co-occurrence methods. It was introduced by Peter Clark et al. at the Allen Institute for AI in 2018. See the paper at arXiv 1803.05457.What does a good ARC-C score look like?
Instruction-tuned models typically score 0.83–0.97. Frontier models now score above 0.96. ARC-C is most useful for comparing smaller models — for frontier comparisons, more challenging benchmarks like GPQA Diamond or MMLU-Pro are better discriminators.Who created ARC-C?
ARC-C was created by Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot and colleagues at the Allen Institute for AI. Published in March 2018 — see arXiv 1803.05457.Is ARC-C saturated?
Yes, for frontier models. The leader scores 97.2% and multiple models exceed 95%, making ARC-C a poor discriminator at the frontier. It remains useful for evaluating smaller or fine-tuned models.How does ARC-C differ from ARC-E?
ARC-C (Challenge Set) contains questions that defeated both retrieval and co-occurrence methods — requiring genuine reasoning. ARC-E (Easy Set) contains questions that at least one such method could answer, making it significantly easier. Both share the same paper and dataset.Benchmark definition paraphrased from Clark et al. 2018. Last updated 2026-07-23.