Video-MMMU — Results
| Rank | Model | Score |
|---|---|---|
| 1 | claude-3-5-sonnet | 65.78 |
| 2 | kimi-vl-a3b-thinking-2506 | 65.22 |
| 3 | gpt-4o | 61.22 |
Video-MMMU
1 phaseActive
300 expert-level lecture videos with 900 questions across Perception, Comprehension, and Adaptation stages, scored with a knowledge-gain (Δknowledge) metric.
Overview
Video-MMMU
Quick answer: Video-MMMU is a multi-modal, multi-disciplinary benchmark that treats educational videos as a source of knowledge rather than just visual content, testing whether large multimodal models can perceive, comprehend, and adapt knowledge from 300 expert-level lecture videos across six professional disciplines — and quantifying genuine learning gain with a novel Δknowledge metric.
At a Glance
What it tests: Knowledge acquisition from college-level lecture videos across three cognitive stages — Perception (identifying key information), Comprehension (understanding underlying concepts), and Adaptation (applying knowledge to solve novel problems) — spanning Art, Business, Science, Medicine, Humanities, and Engineering.
Why it matters: Traditional VideoQA benchmarks mainly test visual content interpretation; Video-MMMU is the first to treat video as an educational knowledge source, directly measuring whether models genuinely learn and improve from watching a video rather than just describing what's in it.
Known limitations: Because it targets college-level, discipline-specific lecture content, strong performance requires substantial domain knowledge alongside video understanding, and the Δknowledge metric's reliability depends on accurately measuring pre-video baseline accuracy on the Adaptation questions.
What Video-MMMU Measures
Video-MMMU comprises 300 lecture-style, college-level videos spanning 30 subjects across 6 professional disciplines (Art, Business, Science, Medicine, Humanities, Engineering), with 900 human-annotated questions — three per video — aligned to three knowledge-acquisition stages: Perception (identifying key information related to the knowledge), Comprehension (understanding the underlying concepts), and Adaptation (applying the knowledge to a new scenario not directly shown in the video).
The benchmark's key innovation is the Δknowledge metric, which quantifies a model's learning gain by comparing its accuracy on Adaptation-track questions before and after watching the relevant instructional video: Δknowledge = (Acc_after − Acc_before) / (100% − Acc_before) × 100%. This shifts evaluation from absolute accuracy toward learning efficiency — can a model actually improve at solving a problem after being shown a video that teaches the relevant concept, the way a human learner would?
The paper's evaluation reveals a steep decline in performance as cognitive demands increase from Perception to Adaptation, and a significant gap between human and model knowledge acquisition: humans achieve Δknowledge of 33.1% after watching the videos, while even top models like GPT-4o (15.6%) and Claude 3.5 Sonnet (11.4%) show substantially smaller learning gains, highlighting a genuine capability gap in how LMMs learn from video-based instruction compared to humans.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Multimodal (video-based knowledge acquisition) |
| Metric | Accuracy (%) per stage (Perception, Comprehension, Adaptation) plus the Δknowledge learning-gain metric |
| Number of tasks | 900 questions (3 per video) across 300 lecture videos in 6 disciplines |
| Saturation | Low — even top models show far smaller knowledge gains than humans (Δknowledge) |
| Created by | Kairui Hu, Penghao Wu, Fanyi Pu, Wang Xiao, Xiang Yue, Bo Li, Yuanhan Zhang, Ziwei Liu |
| Source paper | Hu et al. 2025 (ACL 2026) |
| Code | GitHub |
| Dataset | Hugging Face |
How Video-MMMU Is Scored
Each of the three question types (Perception, Comprehension, Adaptation) is scored as multiple-choice accuracy, and the Adaptation track is additionally used to compute the Δknowledge metric by comparing pre-video and post-video accuracy on matched questions, isolating genuine learning gain from raw knowledge the model already had.
State-of-the-Art Results
| Rank | Model | Overall Accuracy | Δknowledge | Source |
|---|---|---|---|---|
| 1 | Claude-3.5-Sonnet | 65.78 | +11.4 | Video-MMMU leaderboard |
| 2 | Kimi-VL-A3B-Thinking-2506 | 65.22 | +3.5 | Video-MMMU leaderboard |
| 3 | GPT-4o | 61.22 | +15.6 | Video-MMMU leaderboard |
Scores sourced from the official Video-MMMU leaderboard; humans achieve Δknowledge of +33.1 for comparison.
Video-MMMU on Benchgen
No Benchgen results yet — be the first to run Video-MMMU.
Video-MMMU vs Other Benchmarks
| Benchmark | What it tests | Tasks | Saturation |
|---|---|---|---|
| Video-MMMU | Knowledge acquisition from lecture videos, Δknowledge | 900 | low |
| MMBench-Video | Long-form, multi-shot video understanding | 1,998 | low |
| MVBench | 20-task comprehensive video understanding | ~4,000 | medium |
Video-MMMU's Δknowledge metric is unique among video benchmarks in directly measuring learning gain rather than just content comprehension, complementing MVBench and MMBench-Video's broader video-understanding coverage.
Run Video-MMMU on Your Model
Benchgen lets teams run Video-MMMU against their own model or agent versions on demand, tracking per-stage accuracy and the Δknowledge learning-gain metric over time to catch regressions in video-based knowledge acquisition.
Related Models & Benchmarks
Explore related multimodal benchmarks like MMBench-Video and MVBench to compare knowledge-acquisition-focused evaluation with broader video comprehension benchmarks.