GSM8K

GSM8K

Quick answer: GSM8K (Grade School Math 8K) is a dataset of 8,500 linguistically diverse grade-school math word problems created by OpenAI in 2021. The 1,319-problem test split measures a model's ability to perform multi-step arithmetic reasoning expressed in natural language. Frontier models now score above 97%, making it highly saturated as a frontier benchmark.

At a Glance

What it tests: Multi-step arithmetic reasoning on word problems solvable by a mathematically gifted middle-school student — requiring 2–8 reasoning steps and basic operations (addition, subtraction, multiplication, division).

Why it matters: GSM8K was the first benchmark to demonstrate that chain-of-thought prompting dramatically improves LLM math reasoning. It remains a standard baseline for comparing math reasoning across model generations.

Known limitations: Frontier models have essentially saturated GSM8K (97%+), making it unable to differentiate between top models. Newer and harder benchmarks like MATH, AIME 2026, and LiveCodeBench Pro are needed to rank frontier reasoning.

What GSM8K Measures

GSM8K was created to measure whether LLMs can solve the kind of math word problems that require combining basic arithmetic with multi-step logical reasoning. Each problem is written in natural language and has a unique numerical answer. The dataset was specifically designed to be diverse in linguistic style — avoiding the structured "if/then" patterns common in older math datasets — to test genuine arithmetic reasoning rather than template matching.

The benchmark is scored by checking whether the model's final numerical answer exactly matches the gold answer. Models are typically evaluated with chain-of-thought (CoT) prompting, where the model is encouraged to show its work before giving the final answer. Zero-shot CoT ("let's think step by step") and few-shot CoT both substantially improve scores over direct-answer prompting.

Benchmark Specifications

How GSM8K Is Scored

Accuracy is measured as the percentage of test problems where the model produces the correct final numerical answer. Most evaluations use CoT prompting (8-shot or zero-shot); scores vary by prompting strategy. GPT-4 class models consistently reach 97%+, while older or smaller models show more variation. The 97%+ ceiling means GSM8K is no longer suitable for ranking frontier models but remains useful for evaluating mid-tier and smaller models.

State-of-the-Art Results

Scores from published technical reports. Most use 8-shot chain-of-thought unless noted. Results may vary by prompt format.

GSM8K vs Other Benchmarks

GSM8K is the right benchmark for evaluating smaller or mid-tier models on basic arithmetic reasoning. Frontier models should instead be evaluated on MATH or AIME-level tasks.

Run GSM8K on Your Model

Benchgen lets teams run GSM8K against their own model versions, track accuracy over time, and catch regressions before deployment — rather than relying on one-time vendor-reported numbers.

Related Models & Benchmarks

• Top performers: GPT-4o · o1 · DeepSeek-V3
• Related benchmarks: MATH · AIME 2026 · LiveCodeBench

Frequently Asked Questions

Benchmark definition paraphrased from Cobbe et al. 2021. State-of-the-art scores sourced from published technical reports and attributed inline. Last updated 2026-07-23.