MATH

MATH

Quick answer: MATH is a dataset of 12,500 competition mathematics problems (7,500 train, 5,000 test) drawn from AMC 10/12, AIME, and other competitions, created by Hendrycks et al. in 2021. Problems span 7 subjects across 5 difficulty levels. Most benchmark reports use MATH-500, a 500-problem representative subset. Frontier reasoning models now score 90–97% on MATH-500; general-purpose models score 60–78%.

At a Glance

What it tests: Competition-level mathematics including algebra, counting and probability, geometry, intermediate algebra, number theory, prealgebra, and precalculus — requiring multi-step symbolic reasoning.

Why it matters: MATH remains the primary benchmark for separating reasoning models (o1, o3, DeepSeek-R1) from general-purpose models. The jump from GPT-4 (~42%) to o1 (94.8%) on MATH-500 demonstrated the power of chain-of-thought and process reward models.

Known limitations: Scores are highly sensitive to prompting — CoT with tools or code interpreters can add 20+ percentage points. Reasoning models are beginning to approach saturation on MATH-500, shifting attention to harder benchmarks like AIME 2026 and FrontierMath.

What MATH Measures

The MATH dataset was designed to test mathematical problem-solving at the competition level — problems that require genuine symbolic manipulation, proof intuition, and creative reasoning rather than arithmetic pattern matching. Problems are organized into 7 subjects (Algebra, Counting & Probability, Geometry, Intermediate Algebra, Number Theory, Prealgebra, Precalculus) and 5 difficulty levels (Level 1–5), with Level 5 corresponding to the hardest AMC 12 and early AIME problems.

The MATH-500 subset, introduced by Lightman et al. (2023) in the "Let's Verify Step by Step" paper, is a curated 500-problem evaluation set that is now the standard reported metric. MATH-500 spans all subjects and difficulty levels proportionally, making it a reliable proxy for performance on the full test set.

Grading is exact-match on the final answer in LaTeX form (e.g., \frac{3}{4}), which can penalize models that produce equivalent expressions in different formats. Evaluations using code-interpreter tools or majority voting typically score significantly higher than single-pass zero-shot CoT.

Benchmark Specifications

How MATH Is Scored

Accuracy is the percentage of MATH-500 problems where the model's final answer exactly matches the gold answer. Different eval harnesses may use different normalization (e.g., SymPy-based equivalence checks), which can shift scores by several points. Unless otherwise noted, scores below reference either the full 5,000-problem test set or MATH-500 — most modern reports use MATH-500. Tool-augmented evaluations (code interpreter) are sometimes labeled separately.

State-of-the-Art Results

All scores on MATH-500 subset using chain-of-thought prompting unless noted. Scores from published technical reports.

MATH vs Other Benchmarks

MATH is the right benchmark for comparing general frontier models on competition mathematics. For ranking the very best reasoning models, AIME or FrontierMath provide better separation.

Run MATH on Your Model

Benchgen lets teams run MATH against custom model checkpoints, compare CoT vs. tool-augmented performance, and catch mathematical reasoning regressions across versions.

Related Models & Benchmarks

• Top performers: o1 · DeepSeek-V3 · Claude 3.5 Sonnet
• Related benchmarks: GSM8K · AIME 2026 · GPQA Diamond

Frequently Asked Questions

Benchmark definition paraphrased from Hendrycks et al. 2021. State-of-the-art scores sourced from published technical reports and attributed inline. Last updated 2026-07-23.