MMLU-Pro

MMLU-Pro

Quick answer: MMLU-Pro is a harder, reasoning-focused evolution of the original MMLU benchmark, developed by Wang et al. (NeurIPS 2024 Spotlight) at the University of Waterloo. It replaces 4-choice questions with 10-choice questions, removes trivial and noisy items, and covers 12,032 curated problems across 14 academic domains. Frontier models score roughly 16–33% lower on MMLU-Pro than on original MMLU, and Chain-of-Thought reasoning helps significantly — unlike on the original benchmark.

At a Glance

What it tests: Expert-level multi-domain knowledge and multi-step reasoning across 14 academic disciplines, using 10-choice multiple-choice questions drawn from university exams and textbooks.

Why it matters: Original MMLU became saturated as frontier models routinely exceeded 90% accuracy, making it hard to distinguish between leading models. MMLU-Pro restores the discriminative power by raising difficulty, reducing noise, and requiring genuine reasoning rather than pattern matching.

Known limitations: Despite the 10-choice format, SOTA models now approach 90% accuracy, indicating the benchmark is moving toward medium saturation. Domain coverage skews toward Western academic subjects, and performance on individual subjects varies widely — a high overall score may mask gaps in specific areas.

What MMLU-Pro Measures

MMLU-Pro presents a model with a question drawn from one of 14 academic domains — Biology, Business, Chemistry, Computer Science, Economics, Engineering, Health, History, Law, Math, Philosophy, Physics, Psychology, or a general "Others" category — and asks it to select the correct answer from ten options. The wider choice set (up from four in original MMLU) substantially reduces the chance of guessing correctly at random (10% vs 25%), forcing the model to commit to more precise knowledge.

The benchmark was created by curating over 12,000 questions from academic exam preparation materials and university-level textbooks, then filtering out questions that are either trivial (solvable by keyword matching) or noisy (ambiguous wording, incorrect answer keys). This curation step is what makes scores on MMLU-Pro more stable and reproducible: the authors measured only 2% variance across 24 different prompt styles, compared to 4–5% variance on the original MMLU.

A key finding from the paper is that Chain-of-Thought reasoning improves performance on MMLU-Pro but has negligible or negative effects on the original MMLU — confirming that the harder benchmark genuinely tests multi-step reasoning rather than recall. Models that simply retrieve memorised facts perform disproportionately worse on MMLU-Pro than models that can reason through an unfamiliar problem.

Benchmark Specifications

How MMLU-Pro Is Scored

A model's MMLU-Pro score is the percentage of questions answered correctly across all 14 domains. Each question has exactly one correct answer among ten options; there is no partial credit. Results are typically reported as a decimal (0.896) or percentage (89.6%).

The evaluation protocol supports both direct answering and Chain-of-Thought (CoT) answering. CoT consistently outperforms direct answering on MMLU-Pro, particularly on math, science, and engineering questions. Published scores from different sources may differ depending on whether CoT was used, the number of few-shot examples provided, and how answers were extracted from model outputs — making direct comparisons across evaluation frameworks imperfect.

To run the official evaluation, models can be assessed using the scripts in the TIGER-AI-Lab/MMLU-Pro GitHub repo, which supports both local inference and API-based evaluation with multi-threading.

State-of-the-Art Results

Alibaba Cloud's Qwen3 family dominates the top of the leaderboard as of July 2026. Scores may reflect CoT vs direct-answer setups — see the scoring methodology section for details.

| Rank | Model | Score | Date | |---|---|---| | 1 | Qwen3.7 Max | 89.6% | 2026-06 | | 2 | Qwen3.6 Plus | 88.5% | 2026-06 | | 2 | Qwen3.7 Plus | 88.5% | 2026-06 | | 4 | MiniMax M2.1 | 88.0% | 2026-06 | | 5 | Qwen3.5-397B-A17B | 87.8% | 2026-06 | | 6 | DeepSeek-V4-Pro-Max | 87.5% | 2026-06 | | 7 | Kimi K2.5 | 87.1% | 2026-06 | | 8 | ERNIE 5.0 | 87.0% | 2026-06 | | 9 | Nemotron 3 Ultra 550B | 86.8% | 2026-06 | | 10 | Qwen3.5-122B-A10B | 86.7% | 2026-06 |

Scores sourced from llm-stats.com and published technical reports. Last updated 2026-07-14.

MMLU-Pro on Benchgen

No Benchgen results yet — be the first to run MMLU-Pro.

MMLU-Pro vs Other Reasoning Benchmarks

MMLU-Pro sits between the saturated original MMLU and the much harder GPQA Diamond in terms of difficulty. It remains the go-to benchmark for broad academic knowledge evaluation when MMLU is too easy but GPQA is too narrow. The key differentiator is the combination of breadth (14 domains, 12K tasks) and genuine reasoning requirement — a quality that saturated MCQ benchmarks like HellaSwag and ARC no longer provide for frontier models.

Run MMLU-Pro on Your Model

Vendor-reported MMLU-Pro scores may use different prompt formats, CoT strategies, and few-shot counts, making cross-vendor comparisons unreliable without a controlled environment. Benchgen lets your team evaluate any model on MMLU-Pro under a fixed protocol — consistent prompting, reproducible scoring, and version-tracked results — so you can detect regressions between model releases and compare models fairly across providers.

Related Models & Benchmarks

• Top performers: Qwen3.7 Max · Qwen3.6 Plus · DeepSeek-V4-Pro-Max · Kimi K2.5
• Related benchmarks: MMLU · GPQA Diamond · SWE-bench Verified

Frequently Asked Questions

Benchmark definition paraphrased from Wang et al. 2024 (NeurIPS 2024 Spotlight). State-of-the-art scores sourced from llm-stats.com and model documentation, attributed inline. Last updated 2026-07-14.