ARC-AGI

ARC-AGI

Quick answer: ARC-AGI (Abstraction and Reasoning Corpus for AGI), introduced by François Chollet in 2019, tests general fluid intelligence through 400 visual grid transformation tasks where models must infer hidden rules from 2–5 demonstration pairs and apply them to novel grids — requiring abstract pattern recognition without memorization or domain-specific knowledge. GPT-5.5 leads at 95.0%.

At a Glance

What it tests: Abstract visual reasoning — given a handful of input→output grid examples, infer the transformation rule and correctly apply it to a new test input. Tasks require pattern recognition, spatial reasoning, and rule generalisation with minimal prior knowledge.

Why it matters: ARC-AGI was explicitly designed to measure human-like general fluid intelligence rather than crystallised knowledge or memorised patterns. Unlike benchmarks that scale with more training data, ARC-AGI tasks require genuine in-context rule discovery, making it a strong proxy for true generalisation. It became the de facto benchmark for tracking progress toward general-purpose AI reasoning.

Known limitations: With GPT-5.5 at 95% and multiple models above 85%, ARC-AGI (the original 2019 version) is approaching saturation for frontier models. Successors (ARC-AGI v2, ARC-AGI-3) have been introduced to maintain difficulty.

What ARC-AGI Measures

Each ARC-AGI task presents a small number (2–5) of input/output grid pairs that implicitly demonstrate a hidden transformation rule. The model must infer the rule from the examples and then produce the correct output for a test input. Grids are up to 30×30 cells, using 10 discrete colours/symbols. The rules require core knowledge priors that humans naturally possess: object recognition, symmetry, spatial relationships, counting, and simple geometry.

The critical design principle is that each task can be solved with minimal prior knowledge — rules are inferrable from the few examples provided, not from pre-training. This distinguishes ARC-AGI from knowledge-based benchmarks and makes high performance genuinely indicative of reasoning capability rather than memorisation.

The benchmark contains 400 public evaluation tasks plus 400 private test tasks used for the official competition leaderboard. Standard reported scores use the public evaluation set.

Benchmark Specifications

How ARC-AGI Is Scored

A task is counted as solved if the model produces the exact correct output grid for the test input. The score is the percentage of tasks solved out of the total. Models are typically allowed multiple attempts per task (usually 2–3 tries), and the task is counted correct if any attempt is correct. This lenient scoring is intentional — the benchmark tests whether a model can find the right answer, not whether it finds it on the first try.

ARC-AGI on Benchgen

No Benchgen results yet — be the first to run ARC-AGI.

ARC-AGI vs Other Benchmarks

ARC-AGI is the original and most widely cited version; use ARC-AGI v2 or ARC-AGI-3 for meaningful differentiation of current frontier models.

Run ARC-AGI on Your Model

Benchgen lets you run ARC-AGI against your own model configurations — tracking whether a new fine-tune or inference strategy improves abstract reasoning — with version-controlled results that make regressions immediately visible.

Frequently Asked Questions