MMLU

MMLU

Quick answer: MMLU (Massive Multitask Language Understanding) is a 57-subject academic knowledge benchmark of 14,042 multiple-choice questions spanning elementary to expert level, created by Hendrycks et al. (2021). It is the most widely reported knowledge benchmark in AI — virtually every model release includes MMLU scores. Frontier models now score 86–89%, approaching the 89.8% reported expert human average.

At a Glance

What it tests: Broad academic knowledge and understanding across 57 subjects including STEM, humanities, social sciences, and professional fields — ranging from high-school level to professional and graduate-level questions.

Why it matters: MMLU is the standard benchmark for comparing a model's breadth of world knowledge. A high MMLU score signals that a model has absorbed large amounts of structured academic information, making it a reliable correlate of general-purpose usefulness.

Known limitations: MMLU is approaching saturation at the frontier (88–90%), and has been criticized for noisy labels and inconsistent answer quality. MMLU-Pro and MMLU-Redux offer cleaner, harder alternatives.

What MMLU Measures

MMLU covers 57 subjects grouped into four categories: STEM (mathematics, physics, chemistry, biology, computer science), humanities (history, philosophy, law), social sciences (economics, psychology, sociology), and other professional knowledge areas (medicine, finance). Questions span 5 difficulty levels, from elementary to expert-level professional licensing.

Each question is 4-choice multiple choice with a single correct answer. The standard evaluation protocol is 5-shot: the model is given 5 example questions and answers from the same subject before answering the test question. This measures both knowledge recall and few-shot generalization.

MMLU was designed to test the breadth of a model's world knowledge — the hypothesis being that a model that has absorbed vast academic text should score well across all subjects. Because it spans such a wide range, a strong MMLU score signals comprehensive pretraining, while weakness in specific subjects reveals gaps.

Benchmark Specifications

How MMLU Is Scored

Accuracy is reported as the percentage of correctly answered questions in the 5-shot setting across all 57 subjects. The standard evaluation uses chain-of-thought or direct prompting depending on the lab. Some labs report average-of-averages (averaging per-subject accuracy then averaging across subjects), while others report micro-average (total correct / total questions) — this can create small differences between reported numbers. Expert human performance is estimated at 89.8%.

State-of-the-Art Results

Expert human estimate: ~89.8%. All scores 5-shot unless noted. Scores from published technical reports.

MMLU vs Other Benchmarks

For frontier model comparisons, MMLU-Pro provides better discrimination. GPQA Diamond tests depth of expert knowledge rather than breadth.

Run MMLU on Your Model

Benchgen lets teams run MMLU across all 57 subjects on their own models, track subject-level breakdowns, and compare knowledge breadth across model versions.

Related Models & Benchmarks

• Top performers: GPT-4o · Claude 3.5 Sonnet · Llama 3.1 405B
• Related benchmarks: MMLU-Pro · GPQA Diamond · HellaSwag

Frequently Asked Questions

Benchmark definition paraphrased from Hendrycks et al. 2021. State-of-the-art scores sourced from published technical reports and attributed inline. Last updated 2026-07-23.