MMLU-Redux — Results
| Rank | Model | Score |
|---|---|---|
| 1 | qwen3-7-max | 0.95 |
| 2 | qwen3-5-397b-a17b | 0.949 |
| 3 | qwen3-6-plus | 0.945 |
| 4 | qwen3-7-plus | 0.945 |
| 5 | kimi-k2-thinking-0905 | 0.944 |
| 6 | qwen3-5-122b-a10b | 0.94 |
| 7 | qwen3-235b-a22b-thinking-2507 | 0.938 |
| 8 | qwen3-vl-235b-a22b-thinking | 0.937 |
| 9 | qwen3-6-27b | 0.935 |
| 10 | deepseek-r1-0528 | 0.934 |
| 11 | qwen3-5-35b-a3b | 0.933 |
| 12 | qwen3-6-35b-a3b | 0.933 |
| 13 | qwen3-5-27b | 0.932 |
| 14 | qwen3-235b-a22b-instruct-2507 | 0.931 |
| 15 | kimi-k2-instruct-0905 | 0.927 |
| 16 | kimi-k2-instruct | 0.927 |
| 17 | qwen3-next-80b-a3b-thinking | 0.925 |
| 18 | qwen3-vl-235b-a22b-instruct | 0.922 |
| 19 | qwen3-vl-32b-thinking | 0.919 |
| 20 | deepseek-v3-1 | 0.918 |
| 21 | qwen3-5-9b | 0.911 |
| 22 | qwen3-next-80b-a3b-instruct | 0.909 |
| 23 | qwen3-vl-30b-a3b-thinking | 0.909 |
| 24 | qwen3-vl-32b-instruct | 0.898 |
| 25 | longcat-flash-thinking | 0.893 |
MMLU-Redux
1 phaseActive
Re-annotated subset of MMLU fixing ground-truth errors in the original dataset. Provides more reliable evaluation metrics. Metric: accuracy (0–1).
Overview
MMLU-Redux
Quick answer: MMLU-Redux is a re-annotated version of the original MMLU (Massive Multitask Language Understanding) benchmark, created by Gema et al. (2024) to address widespread ground-truth errors found in the original dataset. By manually auditing and correcting labeling errors across many MMLU questions, MMLU-Redux provides more reliable evaluation metrics and reveals that many benchmark scores on the original MMLU were inflated or distorted by noisy labels.
At a Glance
What it tests: The same breadth of academic knowledge as MMLU — 57 subjects spanning STEM, humanities, social sciences, law, medicine, and professional fields — but with corrected ground-truth labels that make the evaluation more trustworthy.
Why it matters: The original MMLU has been widely criticized for containing incorrect answers, ambiguous questions, and annotation errors. Models that appear to score highly on MMLU may be exploiting label noise. MMLU-Redux provides a cleaner signal for comparing model performance.
Known limitations: MMLU-Redux is still derived from the original MMLU problem set, inheriting its scope and difficulty distribution. It does not add new problems, only corrects labels. Frontier models score 90%+ and the benchmark is approaching saturation.
What MMLU-Redux Measures
MMLU-Redux was created by auditing thousands of MMLU questions and identifying cases where the stated correct answer was wrong or ambiguous. The re-annotation process involved domain experts reviewing questions and correcting ground truth labels.
The benchmark covers the same 57 subjects as the original MMLU:
- STEM: mathematics, physics, chemistry, biology, computer science, engineering
- Social Sciences: economics, psychology, sociology, political science
- Humanities: history, philosophy, law, literature
- Professional: medicine, law, accounting, finance
Benchmark Specifications
| Field | Value |
|---|---|
| Tasks | ~3,000 re-annotated questions |
| Subjects | 57 (same as MMLU) |
| Primary metric | Accuracy (% correct) |
| Created by | Gema et al. |
| Paper | arXiv:2406.04127 (Jun 2024) |
| Saturation | High — frontier models 93%+ |
Frequently Asked Questions
How is MMLU-Redux different from MMLU? MMLU-Redux corrects ground-truth errors found in the original MMLU. Many original MMLU questions had incorrect or ambiguous answers; MMLU-Redux provides cleaner labels, making it a more reliable evaluation.
Do models score differently on MMLU-Redux vs original MMLU? Yes — models that scored high on MMLU by pattern-matching to incorrect labels may score lower on MMLU-Redux. The re-annotation also reveals that some models' MMLU scores were inflated by the original label noise.
What score do frontier models get on MMLU-Redux? Top models (Qwen3.7 Max, Qwen3.5-397B-A17B, and similar frontier models) score 93–95% on MMLU-Redux as of mid-2026.