# MME-RealWorld ![Category](https://img.shields.io/badge/Category-Multimodal-1f6feb) ![Metric](https://img.shields.io/badge/Metric-Accuracy-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-29%2C429-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-low-d29922) ![Created](https://img.shields.io/badge/Created-Aug%202024-555) [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/abs/2408.13257) [![GitHub](https://img.shields.io/badge/GitHub-Code-181717?logo=github&logoColor=white)](https://github.com/MME-Benchmarks/MME-RealWorld) > **Quick answer:** MME-RealWorld is the largest fully human-annotated multimodal LLM benchmark to date, featuring 29,429 QA pairs drawn from 13,366 high-resolution images (averaging 2,000×1,500 pixels) across 5 real-world scenarios, specifically designed so that even humans find many tasks difficult. ## At a Glance **What it tests:** High-resolution real-world perception and reasoning across 43 subtasks spanning OCR-in-the-wild, remote sensing, diagrams/tables, video monitoring, and autonomous driving. **Why it matters:** Existing MLLM benchmarks often suffer from small data scale, model-generated (rather than human) annotations, and insufficient difficulty due to limited image resolution; MME-RealWorld addresses all three by using the highest average resolution of any comparable benchmark, fully human annotation from 32 volunteers, and tasks like counting 133 vehicles in a monitoring frame that are genuinely hard even for people. **Known limitations:** High-resolution image processing is computationally demanding for evaluation, and the benchmark is licensed for academic research only with commercial use prohibited. ## What MME-RealWorld Measures MME-RealWorld collects more than 300K images from public datasets and the internet, filtering down to 13,366 high-quality images for annotation by a team of 25 crowdsource workers and 7 MLLM experts, resulting in 29,429 question-answer pairs across 43 subtasks spanning 5 real-world scenarios: OCR in the Wild, Remote Sensing, Diagram and Table, Video Monitoring, and Autonomous Driving. A companion MME-RealWorld-CN adds 5,917 QA pairs using genuinely Chinese-sourced scenarios and volunteer annotation, avoiding the mismatch problems that arise from translating an English-only benchmark. The paper's evaluation of 28 prominent MLLMs, including GPT-4o, Gemini 1.5 Pro, and Claude 3.5 Sonnet, found that none surpassed 60% accuracy — a deliberate design outcome given tasks like counting small objects across a 5000×5000-pixel remote sensing image or tallying dozens of vehicles in a monitoring frame, which challenge both resolution handling and fine-grained counting simultaneously. Because every annotation is manually completed and cross-checked by a professional team, MME-RealWorld avoids the restricted data quality that comes from model-generated annotations used in some other large-scale benchmarks. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Multimodal (high-resolution real-world perception & reasoning) | | Metric | Accuracy (%), reported per scenario/subtask and overall (weighted and unweighted average) | | Number of tasks | 29,429 QA pairs across 43 subtasks and 5 real-world scenarios (13,366 images) | | Saturation | Low — no evaluated model surpassed 60% accuracy at release | | Created by | Yi-Fan Zhang, Huanyu Zhang, Haochen Tian, Chaoyou Fu, and collaborators | | Source paper | [Zhang et al. 2024](https://arxiv.org/abs/2408.13257) (ICLR 2025) | | Code | [GitHub](https://github.com/MME-Benchmarks/MME-RealWorld) | | Dataset | [Hugging Face](https://huggingface.co/datasets/yifanzhang114/MME-RealWorld) | ## How MME-RealWorld Is Scored Each subtask is scored as multiple-choice accuracy without third-party model involvement in evaluation, then reported as both weighted average accuracy (Avg) and unweighted average accuracy across subtasks (Avg-C) for each of the 5 scenario domains (OCR, Remote Sensing, Diagram/Table, Monitoring, Autonomous Driving). ## State-of-the-Art Results > Scores fluctuate by model release; a 10B-parameter model (Awaker2.5-VL-10B) was the first to cross 60% accuracy shortly after release. See the [official leaderboard](https://mme-realworld.github.io/home_page.html#leaderboard) for current standings. ## MME-RealWorld on Benchgen > No Benchgen results yet — [be the first to run MME-RealWorld](/benchmarks/zhang-et-al-xiamen-university/mme-realworld). ## MME-RealWorld vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **MME-RealWorld** | High-resolution real-world perception across 5 scenarios | 29,429 | low | | [MME](/benchmarks/chaoyou-fu-et-al-xiamen-university-tencent-youtu-lab/mme) | 14-subtask perception + cognition, standard resolution | 2,374 | high | | [MMBench-Video](/benchmarks/fang-et-al-shanghai-ai-laboratory/mmbench-video) | Long-form, multi-shot video understanding | 1,998 | low | MME-RealWorld is the direct high-resolution, harder successor to the original MME benchmark, purpose-built to remain unsaturated as frontier models continue to close the gap on MME's original 14 subtasks. ## Run MME-RealWorld on Your Model Benchgen lets teams run the full MME-RealWorld suite against their own model or agent versions on demand, tracking per-scenario accuracy (OCR, remote sensing, monitoring, driving, diagrams) over time to catch regressions specific to high-resolution real-world perception. ## Related Models & Benchmarks Explore related multimodal benchmarks like [MME](/benchmarks/chaoyou-fu-et-al-xiamen-university-tencent-youtu-lab/mme) and [MMBench-Video](/benchmarks/fang-et-al-shanghai-ai-laboratory/mmbench-video) to compare standard-resolution perception and video understanding with MME-RealWorld's high-resolution, real-world focus. ## Frequently Asked Questions

What is MME-RealWorld?

MME-RealWorld is the largest fully human-annotated MLLM benchmark to date, with 29,429 QA pairs from 13,366 high-resolution images across 5 real-world scenarios including OCR, remote sensing, and autonomous driving.

What does a good MME-RealWorld score look like?

At release, no evaluated model (including GPT-4o, Gemini 1.5 Pro, and Claude 3.5 Sonnet) surpassed 60% accuracy, so any model crossing that threshold represents a genuinely strong, state-of-the-art result.

Who created MME-RealWorld?

MME-RealWorld was created by Yi-Fan Zhang and collaborators at Xiamen University and Tencent Youtu Lab, accepted at ICLR 2025.

Rank	Model	Score
1	llava-onevision-7b	57.4
2	qwen2-vl-7b	56.5
3	internvl2-7b	53.5
4	claude-3-5-sonnet	51.6
5	gpt-4o	45.2
6	gemini-1-5-pro	38.2
7	gpt-4o-mini	36.4

MME-RealWorld — Results

MME-RealWorld — Evaluations