MVBench
1 phaseActive
20 challenging video tasks spanning perception to cognition, automatically converted from public video annotations into multiple-choice QA.
Overview
MVBench
Quick answer: MVBench is a comprehensive multimodal video understanding benchmark spanning 20 temporal-related task types — from low-level perception to higher-level cognition — built using a novel static-to-dynamic conversion method that turns public video annotations into multiple-choice QA without heavy manual effort.
At a Glance
What it tests: 20 video tasks that cannot be solved from a single frame, covering a broad spectrum of temporal skills from basic perception (action recognition, object tracking) to cognition-level reasoning (counterfactual inference, episodic reasoning).
Why it matters: Prior multimodal benchmarks focused mainly on static-image spatial understanding; MVBench's static-to-dynamic method systematically generates genuinely temporal tasks at scale, and its accompanying VideoChat2 baseline showed the value of progressive multimodal instruction tuning.
Known limitations: Because tasks are auto-converted from existing public video annotation datasets via multiple-choice QA templates, some tasks may still be partially solvable with strong language priors rather than pure video understanding.
What MVBench Measures
MVBench introduces a static-to-dynamic task-definition method: existing static image/video annotation tasks are systematically transformed into genuinely temporal tasks that require reasoning across multiple frames, covering a broad range of skills from perception (e.g., action recognition, object counting) to cognition (e.g., counterfactual and episodic reasoning). Public video annotations are then automatically converted into multiple-choice QA format, allowing the benchmark to be built efficiently at scale with minimal manual annotation while preserving ground-truth fairness (avoiding LLM-judge scoring bias).
The paper's accompanying VideoChat2 baseline — trained via progressive multimodal instruction tuning on diverse data — surpassed leading contemporary video MLLMs by over 15% on MVBench, demonstrating both the benchmark's discriminative power and the value of the proposed training recipe.
MVBench has since become a standard reference point for video-LLM releases, with continual updates to its evaluation scripts (e.g., a small QA de-duplication fix in mid-2024) tracked in its GitHub repository.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Multimodal (comprehensive video understanding) |
| Metric | Accuracy (%) via multiple-choice QA, averaged across 20 task types |
| Number of tasks | 20 temporal task types, roughly 200 QA pairs each (~4,000 total) |
| Saturation | Medium — strong open models have closed much of the gap since 2023 release |
| Created by | Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, and collaborators |
| Source paper | Li et al. 2023 (CVPR 2024 Highlight) |
| Code | GitHub |
| Dataset | Hugging Face |
How MVBench Is Scored
Each of the 20 task types is scored as multiple-choice QA accuracy, then averaged (typically unweighted) into an overall MVBench score, letting teams see which specific temporal skill (e.g., counterfactual reasoning vs. simple action recognition) a model is weak on.
State-of-the-Art Results
Scores fluctuate by model release and frame-sampling configuration. See the official repository for current standings.
MVBench on Benchgen
No Benchgen results yet — be the first to run MVBench.
MVBench vs Other Benchmarks
| Benchmark | What it tests | Tasks | Saturation |
|---|---|---|---|
| MVBench | 20-task comprehensive video understanding, perception to cognition | ~4,000 | medium |
| TOMATO | Temporal reasoning resistant to single-frame/order shortcuts | 1,484 | low |
| TempCompass | Temporal aspects via conflicting-video pairs | 410 videos | low |
MVBench's 20-task breadth makes it a general-purpose video-understanding yardstick, while TOMATO and TempCompass drill into the narrower, harder problem of proving genuine (shortcut-resistant) temporal reasoning.
Run MVBench on Your Model
Benchgen lets teams run the full 20-task MVBench suite against their own model or agent versions on demand, tracking per-task-type accuracy over time to catch regressions in specific temporal skills that an aggregate score would hide.
Related Models & Benchmarks
Explore related multimodal benchmarks like TOMATO and TempCompass to compare broad video-understanding coverage with narrower temporal-reasoning stress tests.