# ARC-AGI ![Category](https://img.shields.io/badge/Category-Abstract%20Reasoning%20%2F%20Vision-1f6feb) ![Metric](https://img.shields.io/badge/Metric-%25%20Solved-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-400-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-Medium-d29922) ![Created](https://img.shields.io/badge/Created-Nov%202019-555) [![Paper](https://img.shields.io/badge/Paper-arXiv%201911.01547-b31b1b)](https://arxiv.org/abs/1911.01547) [![GitHub](https://img.shields.io/badge/GitHub-fchollet%2FARC--AGI-181717?logo=github&logoColor=white)](https://github.com/fchollet/ARC-AGI) > **Quick answer:** ARC-AGI (Abstraction and Reasoning Corpus for AGI), introduced by François Chollet in 2019, tests general fluid intelligence through 400 visual grid transformation tasks where models must infer hidden rules from 2–5 demonstration pairs and apply them to novel grids — requiring abstract pattern recognition without memorization or domain-specific knowledge. GPT-5.5 leads at 95.0%. ## At a Glance **What it tests:** Abstract visual reasoning — given a handful of input→output grid examples, infer the transformation rule and correctly apply it to a new test input. Tasks require pattern recognition, spatial reasoning, and rule generalisation with minimal prior knowledge. **Why it matters:** ARC-AGI was explicitly designed to measure human-like general fluid intelligence rather than crystallised knowledge or memorised patterns. Unlike benchmarks that scale with more training data, ARC-AGI tasks require genuine in-context rule discovery, making it a strong proxy for true generalisation. It became the de facto benchmark for tracking progress toward general-purpose AI reasoning. **Known limitations:** With GPT-5.5 at 95% and multiple models above 85%, ARC-AGI (the original 2019 version) is approaching saturation for frontier models. Successors (ARC-AGI v2, ARC-AGI-3) have been introduced to maintain difficulty. ## What ARC-AGI Measures Each ARC-AGI task presents a small number (2–5) of input/output grid pairs that implicitly demonstrate a hidden transformation rule. The model must infer the rule from the examples and then produce the correct output for a test input. Grids are up to 30×30 cells, using 10 discrete colours/symbols. The rules require core knowledge priors that humans naturally possess: object recognition, symmetry, spatial relationships, counting, and simple geometry. The critical design principle is that each task can be solved with minimal prior knowledge — rules are inferrable from the few examples provided, not from pre-training. This distinguishes ARC-AGI from knowledge-based benchmarks and makes high performance genuinely indicative of reasoning capability rather than memorisation. The benchmark contains 400 public evaluation tasks plus 400 private test tasks used for the official competition leaderboard. Standard reported scores use the public evaluation set. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Abstract reasoning / Vision | | Metric | % tasks solved | | Number of tasks | 400 (public eval), 400 (private test) | | Grid size | Up to 30×30 | | Colours | 10 | | Saturation | Medium (approaching high for frontier) | | Created by | François Chollet | | Source paper | [Chollet 2019](https://arxiv.org/abs/1911.01547) | | GitHub | [fchollet/ARC-AGI](https://github.com/fchollet/ARC-AGI) | ## How ARC-AGI Is Scored A task is counted as solved if the model produces the exact correct output grid for the test input. The score is the percentage of tasks solved out of the total. Models are typically allowed multiple attempts per task (usually 2–3 tries), and the task is counted correct if any attempt is correct. This lenient scoring is intentional — the benchmark tests whether a model *can* find the right answer, not whether it finds it on the first try. ## ARC-AGI on Benchgen > No Benchgen results yet — [be the first to run ARC-AGI](/benchmarks/francois-chollet/arc-agi). ## ARC-AGI vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **ARC-AGI** | Abstract grid-based rule inference | 400 | Medium | | [ARC-AGI v2](/benchmarks/francois-chollet/arc-agi-v2) | Harder abstract reasoning tasks | 400 | Low | | [ARC-AGI-3](/benchmarks/francois-chollet/arc-agi-3) | Next-generation abstract reasoning | TBD | Low | | [GPQA Diamond](/benchmarks/rein-et-al/gpqa-diamond) | Expert-level science reasoning | 198 | Low | ARC-AGI is the original and most widely cited version; use ARC-AGI v2 or ARC-AGI-3 for meaningful differentiation of current frontier models. ## Run ARC-AGI on Your Model Benchgen lets you run ARC-AGI against your own model configurations — tracking whether a new fine-tune or inference strategy improves abstract reasoning — with version-controlled results that make regressions immediately visible. ## Frequently Asked Questions

What is ARC-AGI?

ARC-AGI (Abstraction and Reasoning Corpus for AGI) is a benchmark of 400 visual grid transformation tasks created by François Chollet in 2019. Each task requires inferring a hidden rule from 2–5 input/output examples and applying it to a new test case, testing general fluid intelligence rather than memorised knowledge.

What does a good score look like on ARC-AGI?

Scores are reported as percentage of tasks solved. Humans score ~84% on the public eval set. As of mid-2026, GPT-5.5 leads at 95.0%, making the original ARC-AGI nearly solved for frontier models. For meaningful differentiation, ARC-AGI v2 (much harder) is now the relevant version.

Who created ARC-AGI?

ARC-AGI was created by François Chollet (creator of Keras) and introduced in the paper "On the Measure of Intelligence" (arXiv:1911.01547, November 2019). It is maintained at github.com/fchollet/ARC-AGI.

Rank	Model	Score
1	gpt-6-astra	98.5
2	gemini-3-1-pro	0.98
3	gpt-5-6-sol	0.975
4	gpt-5-6-terra	0.965
5	gpt-5-5	0.95
6	gpt-5-4	0.937
7	claude-opus-4-7	0.935
8	claude-opus-4-8	0.925
9	gpt-5-2-pro-2025-12-11	0.905
10	grok-4-20	0.895
11	o3	0.88
12	gpt-5-6-luna	0.88
13	gpt-5-2	0.862
14	inkling	0.795
15	glm-5-2	0.77
16	longcat-flash-thinking	0.503
17	qwen3-235b-a22b-instruct-2507	0.418

ARC-AGI — Results

ARC-AGI — Evaluations