MMT-Bench — Results
| Rank | Model | Score |
|---|---|---|
| 1 | internvl2-40b | 66.9 |
| 2 | gpt-4o | 65.4 |
| 3 | gemini-1-5-pro | 64.5 |
MMT-Bench
1 phaseActive
31,325 multi-choice visual questions across 32 meta-tasks and 162 subtasks spanning driving, navigation, and expert-knowledge domains.
Overview
MMT-Bench
Quick answer: MMT-Bench is a large-scale multimodal benchmark with 31,325 multi-choice visual questions spanning 32 core meta-tasks and 162 subtasks — from vehicle driving to embodied navigation — designed to assess LVLMs' progress toward general-purpose multimodal intelligence rather than narrow single-domain capability.
At a Glance
What it tests: Expert-knowledge-requiring multimodal tasks spanning visual recognition, localization, reasoning, and planning across dozens of real-world scenarios (driving, navigation, and beyond).
Why it matters: Most multimodal benchmarks cover a limited slice of tasks; MMT-Bench's task-map structure across 32 meta-tasks lets researchers see both in-domain and out-of-domain capability gaps, tracking genuine multitask generalization rather than a single narrow skill.
Known limitations: With 31,325 questions across 162 subtasks, full-suite evaluation is resource-intensive, which is why the benchmark also ships a smaller curated VAL split for faster iteration.
What MMT-Bench Measures
MMT-Bench comprises 31,325 meticulously curated multi-choice visual questions drawn from diverse multimodal scenarios — vehicle driving, embodied navigation, and many others — covering 32 core meta-tasks and 162 subtasks in multimodal understanding. Because of this extensive task coverage, MMT-Bench enables evaluation via a "task map," letting researchers discover which specific tasks a model handles well (in-domain) versus poorly (out-of-domain) rather than relying on one opaque score.
At release, evaluation of 30 LVLMs including proprietary GPT-4V and GeminiProVision alongside open-sourced InternVL-Chat underscored the significant remaining challenges the benchmark poses, and it has continued to serve as a stress test as newer models like GPT-4o have since taken the top spot on both its VAL and full-set leaderboards.
MMT-Bench is integrated into the widely used VLMEvalKit evaluation framework, making it straightforward for teams to reproduce standardized scores across many open and closed models.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Multimodal (multitask visual recognition, localization, reasoning, planning) |
| Metric | Accuracy (%) via multiple-choice questions |
| Number of tasks | 31,325 questions across 32 meta-tasks and 162 subtasks |
| Saturation | Medium — GPT-4o leads at ~65% on both VAL and full splits, well above random guess (~28-32%) but far from ceiling |
| Created by | Kaining Ying, Fanqing Meng, Jin Wang, Zhiqian Li, and collaborators |
| Source paper | Ying et al. 2024 (ICML 2024) |
| Code | GitHub |
| Dataset | Hugging Face |
How MMT-Bench Is Scored
Each of the 162 subtasks is scored as multiple-choice accuracy, then rolled up into meta-task scores and one overall benchmark score across both the smaller VAL split and the full 31,325-question ALL split, letting teams compare fast iteration (VAL) against comprehensive evaluation (ALL).
State-of-the-Art Results
| Rank | Model | Split | Score | Source |
|---|---|---|---|---|
| 1 | GPT-4o | VAL | 65.4 | MMT-Bench leaderboard |
| 1 | GPT-4o | Full | 65.5 | MMT-Bench leaderboard |
| 2 | InternVL2-40B | VAL | 66.9 | MMT-Bench leaderboard |
Scores sourced from the official MMT-Bench leaderboard; see the repository for the full VAL and ALL split rankings.
MMT-Bench on Benchgen
No Benchgen results yet — be the first to run MMT-Bench.
MMT-Bench vs Other Benchmarks
| Benchmark | What it tests | Tasks | Saturation |
|---|---|---|---|
| MMT-Bench | 32-meta-task, 162-subtask multitask multimodal benchmark | 31,325 | medium |
| MME | 14-subtask perception + cognition | 2,374 | high |
| MMSearch | Multimodal search-engine reasoning | 300 | low |
MMT-Bench's scale and task-map design make it well suited to tracking multitask generalization across many real-world domains, complementing narrower benchmarks like MME (perception/cognition) or MMSearch (search-specific reasoning).
Run MMT-Bench on Your Model
Benchgen lets teams run MMT-Bench (VAL or full split) against their own model or agent versions on demand, tracking per-meta-task accuracy over time to catch domain-specific regressions that an aggregate score would hide.
Related Models & Benchmarks
Explore related multimodal benchmarks like MME and MMSearch to compare broad multitask coverage with narrower perception and search-reasoning evaluations.