# MMT-Bench ![Category](https://img.shields.io/badge/Category-Multimodal-1f6feb) ![Metric](https://img.shields.io/badge/Metric-Accuracy-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-31%2C325-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-medium-d29922) ![Created](https://img.shields.io/badge/Created-Apr%202024-555) [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/abs/2404.16006) [![GitHub](https://img.shields.io/badge/GitHub-Code-181717?logo=github&logoColor=white)](https://github.com/OpenGVLab/MMT-Bench) > **Quick answer:** MMT-Bench is a large-scale multimodal benchmark with 31,325 multi-choice visual questions spanning 32 core meta-tasks and 162 subtasks — from vehicle driving to embodied navigation — designed to assess LVLMs' progress toward general-purpose multimodal intelligence rather than narrow single-domain capability. ## At a Glance **What it tests:** Expert-knowledge-requiring multimodal tasks spanning visual recognition, localization, reasoning, and planning across dozens of real-world scenarios (driving, navigation, and beyond). **Why it matters:** Most multimodal benchmarks cover a limited slice of tasks; MMT-Bench's task-map structure across 32 meta-tasks lets researchers see both in-domain and out-of-domain capability gaps, tracking genuine multitask generalization rather than a single narrow skill. **Known limitations:** With 31,325 questions across 162 subtasks, full-suite evaluation is resource-intensive, which is why the benchmark also ships a smaller curated VAL split for faster iteration. ## What MMT-Bench Measures MMT-Bench comprises 31,325 meticulously curated multi-choice visual questions drawn from diverse multimodal scenarios — vehicle driving, embodied navigation, and many others — covering 32 core meta-tasks and 162 subtasks in multimodal understanding. Because of this extensive task coverage, MMT-Bench enables evaluation via a "task map," letting researchers discover which specific tasks a model handles well (in-domain) versus poorly (out-of-domain) rather than relying on one opaque score. At release, evaluation of 30 LVLMs including proprietary GPT-4V and GeminiProVision alongside open-sourced InternVL-Chat underscored the significant remaining challenges the benchmark poses, and it has continued to serve as a stress test as newer models like GPT-4o have since taken the top spot on both its VAL and full-set leaderboards. MMT-Bench is integrated into the widely used VLMEvalKit evaluation framework, making it straightforward for teams to reproduce standardized scores across many open and closed models. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Multimodal (multitask visual recognition, localization, reasoning, planning) | | Metric | Accuracy (%) via multiple-choice questions | | Number of tasks | 31,325 questions across 32 meta-tasks and 162 subtasks | | Saturation | Medium — GPT-4o leads at ~65% on both VAL and full splits, well above random guess (~28-32%) but far from ceiling | | Created by | Kaining Ying, Fanqing Meng, Jin Wang, Zhiqian Li, and collaborators | | Source paper | [Ying et al. 2024](https://arxiv.org/abs/2404.16006) (ICML 2024) | | Code | [GitHub](https://github.com/OpenGVLab/MMT-Bench) | | Dataset | [Hugging Face](https://huggingface.co/datasets/OpenGVLab/MMT-Bench) | ## How MMT-Bench Is Scored Each of the 162 subtasks is scored as multiple-choice accuracy, then rolled up into meta-task scores and one overall benchmark score across both the smaller VAL split and the full 31,325-question ALL split, letting teams compare fast iteration (VAL) against comprehensive evaluation (ALL). ## State-of-the-Art Results | Rank | Model | Split | Score | Source | |---|---|---|---|---| | 1 | GPT-4o | VAL | 65.4 | [MMT-Bench leaderboard](https://github.com/OpenGVLab/MMT-Bench) | | 1 | GPT-4o | Full | 65.5 | [MMT-Bench leaderboard](https://github.com/OpenGVLab/MMT-Bench) | | 2 | InternVL2-40B | VAL | 66.9 | [MMT-Bench leaderboard](https://github.com/OpenGVLab/MMT-Bench) | *Scores sourced from the official MMT-Bench leaderboard; see the repository for the full VAL and ALL split rankings.* ## MMT-Bench on Benchgen > No Benchgen results yet — [be the first to run MMT-Bench](/benchmarks/ying-et-al-shanghai-ai-laboratory/mmt-bench). ## MMT-Bench vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **MMT-Bench** | 32-meta-task, 162-subtask multitask multimodal benchmark | 31,325 | medium | | [MME](/benchmarks/chaoyou-fu-et-al-xiamen-university-tencent-youtu-lab/mme) | 14-subtask perception + cognition | 2,374 | high | | [MMSearch](/benchmarks/jiang-et-al-cuhk-mmlab/mmsearch) | Multimodal search-engine reasoning | 300 | low | MMT-Bench's scale and task-map design make it well suited to tracking multitask generalization across many real-world domains, complementing narrower benchmarks like MME (perception/cognition) or MMSearch (search-specific reasoning). ## Run MMT-Bench on Your Model Benchgen lets teams run MMT-Bench (VAL or full split) against their own model or agent versions on demand, tracking per-meta-task accuracy over time to catch domain-specific regressions that an aggregate score would hide. ## Related Models & Benchmarks Explore related multimodal benchmarks like [MME](/benchmarks/chaoyou-fu-et-al-xiamen-university-tencent-youtu-lab/mme) and [MMSearch](/benchmarks/jiang-et-al-cuhk-mmlab/mmsearch) to compare broad multitask coverage with narrower perception and search-reasoning evaluations. ## Frequently Asked Questions

What is MMT-Bench?

MMT-Bench is a large-scale multimodal benchmark with 31,325 multi-choice questions across 32 meta-tasks and 162 subtasks, spanning domains like vehicle driving and embodied navigation to assess multitask LVLM capability.

What does a good MMT-Bench score look like?

Leading models like GPT-4o and InternVL2-40B score in the mid-to-high 60s (%) on both the VAL and full splits, well above the ~28-32% random-guess baseline but far from saturation.

Who created MMT-Bench?

MMT-Bench was created by Kaining Ying and collaborators at Shanghai AI Laboratory (OpenGVLab), accepted at ICML 2024.

MMT-Bench — Results

MMT-Bench — Evaluations