MME-RealWorld — Results
| Rank | Model | Score |
|---|---|---|
| 1 | llava-onevision-7b | 57.4 |
| 2 | qwen2-vl-7b | 56.5 |
| 3 | internvl2-7b | 53.5 |
| 4 | claude-3-5-sonnet | 51.6 |
| 5 | gpt-4o | 45.2 |
| 6 | gemini-1-5-pro | 38.2 |
| 7 | gpt-4o-mini | 36.4 |
MME-RealWorld
1 phaseActive
29,429 human-annotated QA pairs across 43 subtasks and 5 real-world scenarios, using the highest-resolution images of any MLLM benchmark to date.
Overview
MME-RealWorld
Quick answer: MME-RealWorld is the largest fully human-annotated multimodal LLM benchmark to date, featuring 29,429 QA pairs drawn from 13,366 high-resolution images (averaging 2,000×1,500 pixels) across 5 real-world scenarios, specifically designed so that even humans find many tasks difficult.
At a Glance
What it tests: High-resolution real-world perception and reasoning across 43 subtasks spanning OCR-in-the-wild, remote sensing, diagrams/tables, video monitoring, and autonomous driving.
Why it matters: Existing MLLM benchmarks often suffer from small data scale, model-generated (rather than human) annotations, and insufficient difficulty due to limited image resolution; MME-RealWorld addresses all three by using the highest average resolution of any comparable benchmark, fully human annotation from 32 volunteers, and tasks like counting 133 vehicles in a monitoring frame that are genuinely hard even for people.
Known limitations: High-resolution image processing is computationally demanding for evaluation, and the benchmark is licensed for academic research only with commercial use prohibited.
What MME-RealWorld Measures
MME-RealWorld collects more than 300K images from public datasets and the internet, filtering down to 13,366 high-quality images for annotation by a team of 25 crowdsource workers and 7 MLLM experts, resulting in 29,429 question-answer pairs across 43 subtasks spanning 5 real-world scenarios: OCR in the Wild, Remote Sensing, Diagram and Table, Video Monitoring, and Autonomous Driving. A companion MME-RealWorld-CN adds 5,917 QA pairs using genuinely Chinese-sourced scenarios and volunteer annotation, avoiding the mismatch problems that arise from translating an English-only benchmark.
The paper's evaluation of 28 prominent MLLMs, including GPT-4o, Gemini 1.5 Pro, and Claude 3.5 Sonnet, found that none surpassed 60% accuracy — a deliberate design outcome given tasks like counting small objects across a 5000×5000-pixel remote sensing image or tallying dozens of vehicles in a monitoring frame, which challenge both resolution handling and fine-grained counting simultaneously.
Because every annotation is manually completed and cross-checked by a professional team, MME-RealWorld avoids the restricted data quality that comes from model-generated annotations used in some other large-scale benchmarks.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Multimodal (high-resolution real-world perception & reasoning) |
| Metric | Accuracy (%), reported per scenario/subtask and overall (weighted and unweighted average) |
| Number of tasks | 29,429 QA pairs across 43 subtasks and 5 real-world scenarios (13,366 images) |
| Saturation | Low — no evaluated model surpassed 60% accuracy at release |
| Created by | Yi-Fan Zhang, Huanyu Zhang, Haochen Tian, Chaoyou Fu, and collaborators |
| Source paper | Zhang et al. 2024 (ICLR 2025) |
| Code | GitHub |
| Dataset | Hugging Face |
How MME-RealWorld Is Scored
Each subtask is scored as multiple-choice accuracy without third-party model involvement in evaluation, then reported as both weighted average accuracy (Avg) and unweighted average accuracy across subtasks (Avg-C) for each of the 5 scenario domains (OCR, Remote Sensing, Diagram/Table, Monitoring, Autonomous Driving).
State-of-the-Art Results
Scores fluctuate by model release; a 10B-parameter model (Awaker2.5-VL-10B) was the first to cross 60% accuracy shortly after release. See the official leaderboard for current standings.
MME-RealWorld on Benchgen
No Benchgen results yet — be the first to run MME-RealWorld.
MME-RealWorld vs Other Benchmarks
| Benchmark | What it tests | Tasks | Saturation |
|---|---|---|---|
| MME-RealWorld | High-resolution real-world perception across 5 scenarios | 29,429 | low |
| MME | 14-subtask perception + cognition, standard resolution | 2,374 | high |
| MMBench-Video | Long-form, multi-shot video understanding | 1,998 | low |
MME-RealWorld is the direct high-resolution, harder successor to the original MME benchmark, purpose-built to remain unsaturated as frontier models continue to close the gap on MME's original 14 subtasks.
Run MME-RealWorld on Your Model
Benchgen lets teams run the full MME-RealWorld suite against their own model or agent versions on demand, tracking per-scenario accuracy (OCR, remote sensing, monitoring, driving, diagrams) over time to catch regressions specific to high-resolution real-world perception.
Related Models & Benchmarks
Explore related multimodal benchmarks like MME and MMBench-Video to compare standard-resolution perception and video understanding with MME-RealWorld's high-resolution, real-world focus.