MMMU-Pro

MMMU-Pro

Quick answer: MMMU-Pro is a harder extension of the MMMU (Massive Multi-discipline Multimodal Understanding) benchmark by Yue et al. (ACL 2025, TIGER-AI-Lab). It filters out questions solvable by text-only models, expands answer choices to 10 options, and includes a vision-only input setting where questions are embedded within images — requiring models to simultaneously read text and reason over visuals. The "Standard 10" setting uses 10 answer choices with standard text+image input. Claude Fable 5 leads the Inkling comparison set at 84.2%.

At a Glance

What it tests: A multimodal model's ability to reason over images and text together across academic disciplines — science, engineering, medicine, arts — using 10-choice questions that cannot be answered from text alone.

Why it matters: Original MMMU can often be solved by text-only models using contextual reasoning without truly "seeing" the images. MMMU-Pro's three-step construction — filter text-solvable questions, expand to 10 choices, embed questions in images — forces genuine visual understanding. Performance drops 16–27% relative to MMMU across models, confirming it tests harder, more realistic multimodal capability.

Known limitations: The "vision-only" setting (text embedded in images) is a novel task format not common in real applications. The 10-choice expansion makes random guessing less likely (10% vs 25%) but also makes the benchmark harder to evaluate without Chain-of-Thought. Scores in the Inkling table are for the "Standard 10" setting only.

What MMMU-Pro Measures

MMMU-Pro is built from MMMU through three steps:

1. Filter: Remove questions that text-only models (without image input) can answer correctly — ensuring all remaining questions require genuine visual understanding.
2. Augment: Expand from 4 to 10 answer choices by adding plausible distractors, reducing the chance of guessing from 25% to 10%.
3. Vision-only setting: Embed the entire question (including text) inside an image, forcing the model to OCR and comprehend the question visually before reasoning. Evaluated separately from the "Standard 10" setting.

The Standard 10 setting used in the Inkling table presents questions in the normal text+image format with 10 answer choices — the primary comparison point across model evaluations.

Topics span 30 academic disciplines including Art, Biology, Business, Chemistry, Computer Science, Economics, Engineering, Finance, Geography, Law, Math, Medicine, Music, Physics, and Psychology.

Benchmark Specifications

How MMMU-Pro Is Scored

Models are evaluated on the Standard 10 setting: each question is presented with its associated image(s) and 10 candidate answers. Scores are reported as % accuracy. Chain-of-Thought prompting generally improves performance; OCR-specific prompts have minimal effect. The vision-only setting (where the question text is embedded inside the image) is scored separately and is not reported in the Inkling comparison table.

State-of-the-Art Results

Scores from Inkling model card (Thinking Machines Lab, July 2026), Standard 10 setting, effort=0.99. Nemotron 3 Ultra, GLM 5.2, and DeepSeek V4 Pro not reported.

MMMU-Pro vs Related Benchmarks

Last updated 2026-07-16.