MME

MME

Quick answer: MME is a comprehensive multimodal LLM benchmark spanning 14 subtasks across two broad ability groups — perception and cognition — with every instruction-answer pair manually written by the authors to prevent data leakage from public datasets.

At a Glance

What it tests: Fine-grained perception (existence, count, position, color, OCR, poster, celebrity, scene, landmark, artwork) and cognition (commonsense reasoning, numerical calculation, text translation, code reasoning) via strict yes/no questions.

Why it matters: MME was one of the first benchmarks to systematically decompose "multimodal understanding" into 14 quantifiable subtasks rather than a single aggregate score, making it easy to spot which specific capability a model is weak in.

Known limitations: The binary yes/no answer format simplifies scoring but is coarser than open-ended VQA, and heavy perception-subtask saturation among frontier models has pushed newer evaluations toward MME's harder successor, MME-RealWorld.

What MME Measures

MME evaluates multimodal LLMs across 14 subtasks split into a perception group (10 subtasks: existence, count, position, color, OCR, poster, celebrity, scene, landmark, artwork) and a cognition group (4 subtasks: commonsense reasoning, numerical calculation, text translation, code reasoning). Every instruction is phrased as a simple yes/no question, and every instruction-answer pair was manually designed by the authors rather than repurposed from existing datasets — a deliberate choice to avoid the risk that a model had already seen the exact question during pretraining.

The paper evaluated 30 advanced MLLMs at release, finding that even leading models had substantial headroom on several cognition subtasks (numerical calculation, code reasoning) despite performing well on straightforward perception tasks like existence and color recognition.

Because each subtask uses a controlled, templated question format, MME makes it straightforward to compare models subtask-by-subtask rather than relying on one opaque aggregate number, which is the basis for its widespread adoption as a standard MLLM evaluation.

Benchmark Specifications

How MME Is Scored

Each subtask contributes an accuracy score (percentage of individual questions answered correctly) and an "Accuracy-Plus" score (percentage of paired questions — one positive, one negative, about the same image — both answered correctly). The 14 subtask scores are then summed into a perception total (out of 2,000) and a cognition total (out of 800), plus an overall combined score, so leaderboards typically report both the perception/cognition split and the grand total.

State-of-the-Art Results

Scores fluctuate by model release cadence and evaluation harness (prompt template, decoding settings). See the official leaderboard for current standings across all 14 subtasks.

MME on Benchgen

No Benchgen results yet — be the first to run MME.

MME vs Other Benchmarks

MME's broad 14-subtask decomposition complements POPE's narrower focus on object hallucination and is the direct predecessor to the harder, higher-resolution MME-RealWorld benchmark.

Run MME on Your Model

Benchgen lets teams run the full MME subtask suite against their own model versions on demand, tracking perception and cognition subtotals over time — catching regressions in specific abilities (like OCR or numerical calculation) that an aggregate score would hide.

Related Models & Benchmarks

Explore related multimodal benchmarks like MME-RealWorld and POPE to compare perception, cognition, and hallucination evaluation approaches.

Frequently Asked Questions