MMBench-Video
1 phaseActive
1,998 QA pairs on ~600 real YouTube videos (30s-6min) spanning 26 fine-grained capabilities, evaluated with a GPT-4-based judge.
Overview
MMBench-Video
Quick answer: MMBench-Video is a long-form, multi-shot video understanding benchmark built from roughly 600 real YouTube videos (30 seconds to 6 minutes long) with 1,998 human-authored question-answer pairs spanning 26 fine-grained capabilities, scored using a GPT-4-based evaluator for closer alignment with human judgment.
At a Glance
What it tests: Holistic long-form video comprehension — including counterfactual reasoning, temporal understanding, and other fine-grained capabilities — across 16 major real-world video categories like News and Sports.
Why it matters: Traditional VideoQA benchmarks often use short clips and templated questions that fail to capture the full spectrum of video content or genuinely assess temporal comprehension; MMBench-Video's free-form questions and GPT-4-based grading better mirror practical video-understanding use cases.
Known limitations: GPT-4-based scoring introduces some judge-model dependency and cost compared to purely rule-based metrics, and the ~600-video scale, while carefully curated, is smaller than some large-scale automatically-constructed video benchmarks.
What MMBench-Video Measures
MMBench-Video incorporates lengthy videos sourced from YouTube and employs free-form questions that mirror practical use cases, rather than relying solely on templated multiple-choice formats. Every question is human-annotated according to a carefully constructed ability taxonomy covering 26 fine-grained capabilities (such as counterfactual reasoning and temporal ordering), specifically designed to probe models' temporal reasoning skills across long, multi-shot videos rather than single, isolated clips.
To score free-form answers reliably, the benchmark employs GPT-4 as an automated judge, which the paper demonstrates achieves superior accuracy and robustness compared to earlier LLM-based evaluation approaches used by prior video benchmarks. The benchmark's comprehensive evaluation spans both proprietary and open-source LVLMs across image and video modalities, providing a unified view of how well current models handle realistic, long-form video content.
MMBench-Video is maintained as part of the OpenCompass VLMEvalKit ecosystem, with an actively updated public leaderboard tracking the latest proprietary and open-source models.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Multimodal (long-form, multi-shot video understanding) |
| Metric | GPT-4-based automated scoring of free-form answers |
| Number of tasks | 1,998 QA pairs on ~600 videos (30s-6min) across 26 capabilities |
| Saturation | Low — long-form, free-form video understanding remains challenging for current LVLMs |
| Created by | Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, Kai Chen |
| Source paper | Fang et al. 2024 (NeurIPS 2024 D&B Track) |
| Code | GitHub |
| Dataset | Hugging Face |
How MMBench-Video Is Scored
Each free-form model answer is graded by a GPT-4-based judge against the human-written reference answer, typically producing a score per question that is then aggregated per capability dimension and into an overall benchmark score, allowing teams to see which of the 26 fine-grained capabilities a model struggles with most.
State-of-the-Art Results
Scores vary by frame-sampling rate (e.g., 8-frame vs. 1fps) and evaluation packing strategy. See the official leaderboard for current standings.
MMBench-Video on Benchgen
No Benchgen results yet — be the first to run MMBench-Video.
MMBench-Video vs Other Benchmarks
| Benchmark | What it tests | Tasks | Saturation |
|---|---|---|---|
| MMBench-Video | Long-form, multi-shot video understanding, GPT-4 judged | 1,998 | low |
| MVBench | 20-task comprehensive video understanding | ~4,000 | medium |
| MME-RealWorld | High-resolution real-world perception & reasoning | 29,429 | low |
MMBench-Video's free-form, long-form questions complement MVBench's structured 20-task multiple-choice design, offering a more open-ended read on genuine video comprehension.
Run MMBench-Video on Your Model
Benchgen lets teams run MMBench-Video against their own model or agent versions on demand, tracking GPT-4-judged scores across all 26 fine-grained capabilities to catch regressions in long-form video comprehension that shorter benchmarks would miss.
Related Models & Benchmarks
Explore related multimodal benchmarks like MVBench and MME-RealWorld to compare structured video-task evaluation with MMBench-Video's free-form, long-form focus.