MATH-500

MATH-500

Quick answer: MATH-500 is a carefully curated 500-problem subset of the full MATH benchmark, created by Lightman et al. at OpenAI as part of the "Let's Verify Step by Step" paper (2023) on process reward models. It covers all 7 difficulty levels and 7 mathematical subjects (algebra, counting & probability, geometry, number theory, pre-algebra, pre-calculus, intermediate algebra), with a representative distribution that makes it a practical, efficient stand-in for the full 12,500-problem MATH benchmark.

At a Glance

What it tests: Mathematical reasoning across 7 subject areas at 5 difficulty levels. Problems are drawn from high school math competitions and require multi-step derivations with exact final answers.

Why it matters: MATH-500 is widely used because it is computationally efficient (500 vs 12,500 problems), statistically reliable (better than AIME's 30 problems), and representative of the full MATH distribution. It became a standard benchmark for evaluating reasoning models and chain-of-thought methods.

Known limitations: Increasingly saturated — frontier models score 90%+ on MATH-500. The subset selection may favor certain problem types. Like the full MATH dataset, problems are publicly available and risk contamination.

What MATH-500 Measures

MATH-500 is drawn from the Hendrycks MATH benchmark (Hendrycks et al., 2021) with 500 problems selected to be representative across all subject-level and difficulty-level buckets. It was introduced in "Let's Verify Step by Step" (Lightman et al., 2023) to enable efficient evaluation of process reward models.

Subject areas covered:

• Algebra — polynomials, functions, equations
• Counting & Probability — combinatorics, discrete probability
• Geometry — Euclidean and coordinate geometry
• Number Theory — divisibility, modular arithmetic, primes
• Pre-Algebra — arithmetic fundamentals
• Pre-Calculus — sequences, limits, trigonometry
• Intermediate Algebra — complex numbers, logs, polynomials

Benchmark Specifications

Frequently Asked Questions

How does MATH-500 relate to the full MATH benchmark? MATH-500 is a 500-problem representative subset of the full MATH benchmark (12,500 problems). It was selected to maintain the subject and difficulty distribution of the full set, making it a reliable proxy that is 25x faster to evaluate.

What is the difference between MATH-500 and AIME? MATH-500 covers a broad range of high school math at difficulty levels 1–5. AIME focuses exclusively on the hardest competition math (~level 5). MATH-500 is better for measuring overall math capability; AIME is better for discriminating among top-tier reasoning models.

What score do frontier models achieve on MATH-500? As of 2025–2026, top models (o3, GPT-5, Gemini 3 Pro) score 95–99% on MATH-500. The benchmark is now largely saturated for frontier models, though it remains useful for tracking mid-tier and specialized models.