# MME ![Category](https://img.shields.io/badge/Category-Multimodal-1f6feb) ![Metric](https://img.shields.io/badge/Metric-Accuracy%20%2B%20Acc--Plus-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-2%2C374-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-high-d29922) ![Created](https://img.shields.io/badge/Created-Jun%202023-555) [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/abs/2306.13394) [![GitHub](https://img.shields.io/badge/GitHub-Project%20Page-181717?logo=github&logoColor=white)](https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models) > **Quick answer:** MME is a comprehensive multimodal LLM benchmark spanning 14 subtasks across two broad ability groups — perception and cognition — with every instruction-answer pair manually written by the authors to prevent data leakage from public datasets. ## At a Glance **What it tests:** Fine-grained perception (existence, count, position, color, OCR, poster, celebrity, scene, landmark, artwork) and cognition (commonsense reasoning, numerical calculation, text translation, code reasoning) via strict yes/no questions. **Why it matters:** MME was one of the first benchmarks to systematically decompose "multimodal understanding" into 14 quantifiable subtasks rather than a single aggregate score, making it easy to spot which specific capability a model is weak in. **Known limitations:** The binary yes/no answer format simplifies scoring but is coarser than open-ended VQA, and heavy perception-subtask saturation among frontier models has pushed newer evaluations toward MME's harder successor, MME-RealWorld. ## What MME Measures MME evaluates multimodal LLMs across 14 subtasks split into a **perception** group (10 subtasks: existence, count, position, color, OCR, poster, celebrity, scene, landmark, artwork) and a **cognition** group (4 subtasks: commonsense reasoning, numerical calculation, text translation, code reasoning). Every instruction is phrased as a simple yes/no question, and every instruction-answer pair was manually designed by the authors rather than repurposed from existing datasets — a deliberate choice to avoid the risk that a model had already seen the exact question during pretraining. The paper evaluated 30 advanced MLLMs at release, finding that even leading models had substantial headroom on several cognition subtasks (numerical calculation, code reasoning) despite performing well on straightforward perception tasks like existence and color recognition. Because each subtask uses a controlled, templated question format, MME makes it straightforward to compare models subtask-by-subtask rather than relying on one opaque aggregate number, which is the basis for its widespread adoption as a standard MLLM evaluation. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Multimodal (perception + cognition) | | Metric | Accuracy and "Accuracy-Plus" (both individual and paired yes/no questions correct), summed per subtask and overall | | Number of tasks | 2,374 manually annotated instruction-answer pairs across 14 subtasks | | Saturation | High — perception subtasks are largely saturated by frontier models; cognition subtasks still show headroom | | Created by | Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, Rongrong Ji, Caifeng Shan, Ran He | | Source paper | [Fu et al. 2023](https://arxiv.org/abs/2306.13394) | | Project page | [GitHub](https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models) | ## How MME Is Scored Each subtask contributes an accuracy score (percentage of individual questions answered correctly) and an "Accuracy-Plus" score (percentage of paired questions — one positive, one negative, about the same image — both answered correctly). The 14 subtask scores are then summed into a perception total (out of 2,000) and a cognition total (out of 800), plus an overall combined score, so leaderboards typically report both the perception/cognition split and the grand total. ## State-of-the-Art Results > Scores fluctuate by model release cadence and evaluation harness (prompt template, decoding settings). See the [official leaderboard](https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models) for current standings across all 14 subtasks. ## MME on Benchgen > No Benchgen results yet — [be the first to run MME](/benchmarks/chaoyou-fu-et-al-xiamen-university-tencent-youtu-lab/mme). ## MME vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **MME** | 14-subtask perception + cognition yes/no evaluation | 2,374 | high | | [MME-RealWorld](/benchmarks/mme-realworld) | Harder, high-resolution real-world successor to MME | — | low | | [POPE](/benchmarks/pope) | Object hallucination via yes/no polling | — | high | MME's broad 14-subtask decomposition complements POPE's narrower focus on object hallucination and is the direct predecessor to the harder, higher-resolution MME-RealWorld benchmark. ## Run MME on Your Model Benchgen lets teams run the full MME subtask suite against their own model versions on demand, tracking perception and cognition subtotals over time — catching regressions in specific abilities (like OCR or numerical calculation) that an aggregate score would hide. ## Related Models & Benchmarks Explore related multimodal benchmarks like [MME-RealWorld](/benchmarks/mme-realworld) and [POPE](/benchmarks/pope) to compare perception, cognition, and hallucination evaluation approaches. ## Frequently Asked Questions

What is MME?

MME is a comprehensive multimodal LLM benchmark with 2,374 manually annotated yes/no instruction-answer pairs spanning 14 subtasks across perception and cognition abilities.

What does a good MME score look like?

Strong frontier models typically score near the perception ceiling (close to 2,000 out of 2,000) but show more variance on the cognition subtotal (out of 800), particularly on numerical calculation and code reasoning.

Who created MME?

MME was created by Chaoyou Fu and collaborators at Xiamen University and Tencent Youtu Lab, with the paper later recognized as a NeurIPS 2025 Datasets & Benchmarks Spotlight.

Rank	Model	Score
1	qwen-vl-max	2433.61
2	minicpm-v-2-6	2365.89
3	internvl-chat-v1-5	2187.84
4	llava-1-6-34b	2028.61
5	gemini-pro	1933.36
6	gpt-4v	1926.57
7	cogvlm	1752.28

MME — Results

MME — Evaluations