# Video-MMMU ![Category](https://img.shields.io/badge/Category-Multimodal-1f6feb) ![Metric](https://img.shields.io/badge/Metric-Accuracy%20%2B%20%CE%94knowledge-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-900-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-low-d29922) ![Created](https://img.shields.io/badge/Created-Jan%202025-555) [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/abs/2501.13826) [![GitHub](https://img.shields.io/badge/GitHub-Code-181717?logo=github&logoColor=white)](https://github.com/EvolvingLMMs-Lab/VideoMMMU) > **Quick answer:** Video-MMMU is a multi-modal, multi-disciplinary benchmark that treats educational videos as a source of knowledge rather than just visual content, testing whether large multimodal models can perceive, comprehend, and adapt knowledge from 300 expert-level lecture videos across six professional disciplines — and quantifying genuine learning gain with a novel Δknowledge metric. ## At a Glance **What it tests:** Knowledge acquisition from college-level lecture videos across three cognitive stages — Perception (identifying key information), Comprehension (understanding underlying concepts), and Adaptation (applying knowledge to solve novel problems) — spanning Art, Business, Science, Medicine, Humanities, and Engineering. **Why it matters:** Traditional VideoQA benchmarks mainly test visual content interpretation; Video-MMMU is the first to treat video as an educational knowledge source, directly measuring whether models genuinely learn and improve from watching a video rather than just describing what's in it. **Known limitations:** Because it targets college-level, discipline-specific lecture content, strong performance requires substantial domain knowledge alongside video understanding, and the Δknowledge metric's reliability depends on accurately measuring pre-video baseline accuracy on the Adaptation questions. ## What Video-MMMU Measures Video-MMMU comprises 300 lecture-style, college-level videos spanning 30 subjects across 6 professional disciplines (Art, Business, Science, Medicine, Humanities, Engineering), with 900 human-annotated questions — three per video — aligned to three knowledge-acquisition stages: Perception (identifying key information related to the knowledge), Comprehension (understanding the underlying concepts), and Adaptation (applying the knowledge to a new scenario not directly shown in the video). The benchmark's key innovation is the Δknowledge metric, which quantifies a model's learning gain by comparing its accuracy on Adaptation-track questions before and after watching the relevant instructional video: Δknowledge = (Acc_after − Acc_before) / (100% − Acc_before) × 100%. This shifts evaluation from absolute accuracy toward learning efficiency — can a model actually improve at solving a problem after being shown a video that teaches the relevant concept, the way a human learner would? The paper's evaluation reveals a steep decline in performance as cognitive demands increase from Perception to Adaptation, and a significant gap between human and model knowledge acquisition: humans achieve Δknowledge of 33.1% after watching the videos, while even top models like GPT-4o (15.6%) and Claude 3.5 Sonnet (11.4%) show substantially smaller learning gains, highlighting a genuine capability gap in how LMMs learn from video-based instruction compared to humans. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Multimodal (video-based knowledge acquisition) | | Metric | Accuracy (%) per stage (Perception, Comprehension, Adaptation) plus the Δknowledge learning-gain metric | | Number of tasks | 900 questions (3 per video) across 300 lecture videos in 6 disciplines | | Saturation | Low — even top models show far smaller knowledge gains than humans (Δknowledge) | | Created by | Kairui Hu, Penghao Wu, Fanyi Pu, Wang Xiao, Xiang Yue, Bo Li, Yuanhan Zhang, Ziwei Liu | | Source paper | [Hu et al. 2025](https://arxiv.org/abs/2501.13826) (ACL 2026) | | Code | [GitHub](https://github.com/EvolvingLMMs-Lab/VideoMMMU) | | Dataset | [Hugging Face](https://huggingface.co/datasets/lmms-lab/VideoMMMU) | ## How Video-MMMU Is Scored Each of the three question types (Perception, Comprehension, Adaptation) is scored as multiple-choice accuracy, and the Adaptation track is additionally used to compute the Δknowledge metric by comparing pre-video and post-video accuracy on matched questions, isolating genuine learning gain from raw knowledge the model already had. ## State-of-the-Art Results | Rank | Model | Overall Accuracy | Δknowledge | Source | |---|---|---|---|---| | 1 | Claude-3.5-Sonnet | 65.78 | +11.4 | [Video-MMMU leaderboard](https://videommmu.github.io/) | | 2 | Kimi-VL-A3B-Thinking-2506 | 65.22 | +3.5 | [Video-MMMU leaderboard](https://videommmu.github.io/) | | 3 | GPT-4o | 61.22 | +15.6 | [Video-MMMU leaderboard](https://videommmu.github.io/) | *Scores sourced from the official Video-MMMU leaderboard; humans achieve Δknowledge of +33.1 for comparison.* ## Video-MMMU on Benchgen > No Benchgen results yet — [be the first to run Video-MMMU](/benchmarks/hu-et-al-nanyang-technological-university/video-mmmu). ## Video-MMMU vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **Video-MMMU** | Knowledge acquisition from lecture videos, Δknowledge | 900 | low | | [MMBench-Video](/benchmarks/fang-et-al-shanghai-ai-laboratory/mmbench-video) | Long-form, multi-shot video understanding | 1,998 | low | | [MVBench](/benchmarks/li-et-al-shanghai-ai-laboratory/mvbench) | 20-task comprehensive video understanding | ~4,000 | medium | Video-MMMU's Δknowledge metric is unique among video benchmarks in directly measuring learning gain rather than just content comprehension, complementing MVBench and MMBench-Video's broader video-understanding coverage. ## Run Video-MMMU on Your Model Benchgen lets teams run Video-MMMU against their own model or agent versions on demand, tracking per-stage accuracy and the Δknowledge learning-gain metric over time to catch regressions in video-based knowledge acquisition. ## Related Models & Benchmarks Explore related multimodal benchmarks like [MMBench-Video](/benchmarks/fang-et-al-shanghai-ai-laboratory/mmbench-video) and [MVBench](/benchmarks/li-et-al-shanghai-ai-laboratory/mvbench) to compare knowledge-acquisition-focused evaluation with broader video comprehension benchmarks. ## Frequently Asked Questions

What is Video-MMMU?

Video-MMMU is a multi-disciplinary benchmark of 300 expert-level lecture videos and 900 questions testing whether LMMs can perceive, comprehend, and adapt knowledge from educational videos, using a novel Δknowledge metric to measure genuine learning gain.

What does a good Video-MMMU score look like?

Top models like Claude-3.5-Sonnet score around 65-66% overall accuracy, but the more telling metric is Δknowledge — humans achieve +33.1%, while even the best models (GPT-4o at +15.6%) show notably smaller learning gains.

Who created Video-MMMU?

Video-MMMU was created by Kairui Hu and collaborators at Nanyang Technological University's S-Lab and Carnegie Mellon University, accepted at ACL 2026.

Video-MMMU — Results

Video-MMMU — Phases

Video-MMMU