# MMBench-Video ![Category](https://img.shields.io/badge/Category-Multimodal-1f6feb) ![Metric](https://img.shields.io/badge/Metric-GPT--4%20Judged%20Score-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-1%2C998-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-low-d29922) ![Created](https://img.shields.io/badge/Created-Jun%202024-555) [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/abs/2406.14515) [![GitHub](https://img.shields.io/badge/GitHub-VLMEvalKit-181717?logo=github&logoColor=white)](https://github.com/open-compass/VLMEvalKit) > **Quick answer:** MMBench-Video is a long-form, multi-shot video understanding benchmark built from roughly 600 real YouTube videos (30 seconds to 6 minutes long) with 1,998 human-authored question-answer pairs spanning 26 fine-grained capabilities, scored using a GPT-4-based evaluator for closer alignment with human judgment. ## At a Glance **What it tests:** Holistic long-form video comprehension — including counterfactual reasoning, temporal understanding, and other fine-grained capabilities — across 16 major real-world video categories like News and Sports. **Why it matters:** Traditional VideoQA benchmarks often use short clips and templated questions that fail to capture the full spectrum of video content or genuinely assess temporal comprehension; MMBench-Video's free-form questions and GPT-4-based grading better mirror practical video-understanding use cases. **Known limitations:** GPT-4-based scoring introduces some judge-model dependency and cost compared to purely rule-based metrics, and the ~600-video scale, while carefully curated, is smaller than some large-scale automatically-constructed video benchmarks. ## What MMBench-Video Measures MMBench-Video incorporates lengthy videos sourced from YouTube and employs free-form questions that mirror practical use cases, rather than relying solely on templated multiple-choice formats. Every question is human-annotated according to a carefully constructed ability taxonomy covering 26 fine-grained capabilities (such as counterfactual reasoning and temporal ordering), specifically designed to probe models' temporal reasoning skills across long, multi-shot videos rather than single, isolated clips. To score free-form answers reliably, the benchmark employs GPT-4 as an automated judge, which the paper demonstrates achieves superior accuracy and robustness compared to earlier LLM-based evaluation approaches used by prior video benchmarks. The benchmark's comprehensive evaluation spans both proprietary and open-source LVLMs across image and video modalities, providing a unified view of how well current models handle realistic, long-form video content. MMBench-Video is maintained as part of the OpenCompass VLMEvalKit ecosystem, with an actively updated public leaderboard tracking the latest proprietary and open-source models. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Multimodal (long-form, multi-shot video understanding) | | Metric | GPT-4-based automated scoring of free-form answers | | Number of tasks | 1,998 QA pairs on ~600 videos (30s-6min) across 26 capabilities | | Saturation | Low — long-form, free-form video understanding remains challenging for current LVLMs | | Created by | Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, Kai Chen | | Source paper | [Fang et al. 2024](https://arxiv.org/abs/2406.14515) (NeurIPS 2024 D&B Track) | | Code | [GitHub](https://github.com/open-compass/VLMEvalKit) | | Dataset | [Hugging Face](https://huggingface.co/datasets/opencompass/MMBench-Video) | ## How MMBench-Video Is Scored Each free-form model answer is graded by a GPT-4-based judge against the human-written reference answer, typically producing a score per question that is then aggregated per capability dimension and into an overall benchmark score, allowing teams to see which of the 26 fine-grained capabilities a model struggles with most. ## State-of-the-Art Results > Scores vary by frame-sampling rate (e.g., 8-frame vs. 1fps) and evaluation packing strategy. See the [official leaderboard](https://huggingface.co/spaces/opencompass/openvlm_video_leaderboard) for current standings. ## MMBench-Video on Benchgen > No Benchgen results yet — [be the first to run MMBench-Video](/benchmarks/fang-et-al-shanghai-ai-laboratory/mmbench-video). ## MMBench-Video vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **MMBench-Video** | Long-form, multi-shot video understanding, GPT-4 judged | 1,998 | low | | [MVBench](/benchmarks/li-et-al-shanghai-ai-laboratory/mvbench) | 20-task comprehensive video understanding | ~4,000 | medium | | [MME-RealWorld](/benchmarks/zhang-et-al-xiamen-university/mme-realworld) | High-resolution real-world perception & reasoning | 29,429 | low | MMBench-Video's free-form, long-form questions complement MVBench's structured 20-task multiple-choice design, offering a more open-ended read on genuine video comprehension. ## Run MMBench-Video on Your Model Benchgen lets teams run MMBench-Video against their own model or agent versions on demand, tracking GPT-4-judged scores across all 26 fine-grained capabilities to catch regressions in long-form video comprehension that shorter benchmarks would miss. ## Related Models & Benchmarks Explore related multimodal benchmarks like [MVBench](/benchmarks/li-et-al-shanghai-ai-laboratory/mvbench) and [MME-RealWorld](/benchmarks/zhang-et-al-xiamen-university/mme-realworld) to compare structured video-task evaluation with MMBench-Video's free-form, long-form focus. ## Frequently Asked Questions

What is MMBench-Video?

MMBench-Video is a long-form, multi-shot video understanding benchmark with 1,998 free-form question-answer pairs on ~600 real YouTube videos, scored with a GPT-4-based automated judge across 26 fine-grained capabilities.

What does a good MMBench-Video score look like?

Because scoring is GPT-4-judged on a per-question basis and aggregated, comparisons are best made against the official OpenVLM video leaderboard rather than a fixed universal threshold, as scores shift with judge-model version and frame-sampling settings.

Who created MMBench-Video?

MMBench-Video was created by Xinyu Fang and collaborators at Shanghai AI Laboratory, accepted at the NeurIPS 2024 Datasets and Benchmarks Track.

MMBench-Video — Phases

MMBench-Video