# MVBench ![Category](https://img.shields.io/badge/Category-Multimodal-1f6feb) ![Metric](https://img.shields.io/badge/Metric-Accuracy-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-20%20task%20types-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-medium-d29922) ![Created](https://img.shields.io/badge/Created-Nov%202023-555) [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/abs/2311.17005) [![GitHub](https://img.shields.io/badge/GitHub-Code-181717?logo=github&logoColor=white)](https://github.com/OpenGVLab/Ask-Anything) > **Quick answer:** MVBench is a comprehensive multimodal video understanding benchmark spanning 20 temporal-related task types — from low-level perception to higher-level cognition — built using a novel static-to-dynamic conversion method that turns public video annotations into multiple-choice QA without heavy manual effort. ## At a Glance **What it tests:** 20 video tasks that cannot be solved from a single frame, covering a broad spectrum of temporal skills from basic perception (action recognition, object tracking) to cognition-level reasoning (counterfactual inference, episodic reasoning). **Why it matters:** Prior multimodal benchmarks focused mainly on static-image spatial understanding; MVBench's static-to-dynamic method systematically generates genuinely temporal tasks at scale, and its accompanying VideoChat2 baseline showed the value of progressive multimodal instruction tuning. **Known limitations:** Because tasks are auto-converted from existing public video annotation datasets via multiple-choice QA templates, some tasks may still be partially solvable with strong language priors rather than pure video understanding. ## What MVBench Measures MVBench introduces a static-to-dynamic task-definition method: existing static image/video annotation tasks are systematically transformed into genuinely temporal tasks that require reasoning across multiple frames, covering a broad range of skills from perception (e.g., action recognition, object counting) to cognition (e.g., counterfactual and episodic reasoning). Public video annotations are then automatically converted into multiple-choice QA format, allowing the benchmark to be built efficiently at scale with minimal manual annotation while preserving ground-truth fairness (avoiding LLM-judge scoring bias). The paper's accompanying VideoChat2 baseline — trained via progressive multimodal instruction tuning on diverse data — surpassed leading contemporary video MLLMs by over 15% on MVBench, demonstrating both the benchmark's discriminative power and the value of the proposed training recipe. MVBench has since become a standard reference point for video-LLM releases, with continual updates to its evaluation scripts (e.g., a small QA de-duplication fix in mid-2024) tracked in its GitHub repository. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Multimodal (comprehensive video understanding) | | Metric | Accuracy (%) via multiple-choice QA, averaged across 20 task types | | Number of tasks | 20 temporal task types, roughly 200 QA pairs each (~4,000 total) | | Saturation | Medium — strong open models have closed much of the gap since 2023 release | | Created by | Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, and collaborators | | Source paper | [Li et al. 2023](https://arxiv.org/abs/2311.17005) (CVPR 2024 Highlight) | | Code | [GitHub](https://github.com/OpenGVLab/Ask-Anything) | | Dataset | [Hugging Face](https://huggingface.co/datasets/OpenGVLab/MVBench) | ## How MVBench Is Scored Each of the 20 task types is scored as multiple-choice QA accuracy, then averaged (typically unweighted) into an overall MVBench score, letting teams see which specific temporal skill (e.g., counterfactual reasoning vs. simple action recognition) a model is weak on. ## State-of-the-Art Results > Scores fluctuate by model release and frame-sampling configuration. See the [official repository](https://github.com/OpenGVLab/Ask-Anything/blob/main/video_chat2/MVBENCH.md) for current standings. ## MVBench on Benchgen > No Benchgen results yet — [be the first to run MVBench](/benchmarks/li-et-al-shanghai-ai-laboratory/mvbench). ## MVBench vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **MVBench** | 20-task comprehensive video understanding, perception to cognition | ~4,000 | medium | | [TOMATO](/benchmarks/shangguan-et-al-yale-university/tomato) | Temporal reasoning resistant to single-frame/order shortcuts | 1,484 | low | | [TempCompass](/benchmarks/liu-et-al-peking-university/tempcompass) | Temporal aspects via conflicting-video pairs | 410 videos | low | MVBench's 20-task breadth makes it a general-purpose video-understanding yardstick, while TOMATO and TempCompass drill into the narrower, harder problem of proving genuine (shortcut-resistant) temporal reasoning. ## Run MVBench on Your Model Benchgen lets teams run the full 20-task MVBench suite against their own model or agent versions on demand, tracking per-task-type accuracy over time to catch regressions in specific temporal skills that an aggregate score would hide. ## Related Models & Benchmarks Explore related multimodal benchmarks like [TOMATO](/benchmarks/shangguan-et-al-yale-university/tomato) and [TempCompass](/benchmarks/liu-et-al-peking-university/tempcompass) to compare broad video-understanding coverage with narrower temporal-reasoning stress tests. ## Frequently Asked Questions

What is MVBench?

MVBench is a comprehensive 20-task video understanding benchmark covering temporal skills from perception to cognition, built via a static-to-dynamic method that converts public video annotations into multiple-choice QA.

What does a good MVBench score look like?

The paper's VideoChat2 baseline surpassed prior leading video MLLMs by over 15%, and top open models have continued to close the gap since — strong current scores tend to be well above 60% average accuracy across all 20 tasks.

Who created MVBench?

MVBench was created by Kunchang Li and collaborators at Shanghai AI Laboratory (OpenGVLab) and Nanjing University, presented as a CVPR 2024 Highlight.

MVBench — Phases

MVBench