# MMVU ![Category](https://img.shields.io/badge/Category-Video%20Understanding-1f6feb) ![Metric](https://img.shields.io/badge/Metric-%25%20Accuracy-2da44e) ![Questions](https://img.shields.io/badge/Questions-3%2C000-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-Low-d29922) ![Created](https://img.shields.io/badge/Created-Jan%202025-555) [![Paper](https://img.shields.io/badge/Paper-arXiv%202501.12380-b31b1b)](https://arxiv.org/abs/2501.12380) > **Quick answer:** MMVU (Zhao et al., Yale NLP, 2025) is a comprehensive expert-level, multi-discipline benchmark for video understanding, with 3,000 expert-annotated questions spanning 27 subjects across Science, Healthcare, Humanities & Social Sciences, and Engineering. Each example includes expert-annotated reasoning rationales and relevant domain knowledge. Kimi K3 scores 82.1% as of July 2026. ## At a Glance **What it tests:** Domain-specific expert knowledge combined with video understanding — models must analyze specialized-domain videos (e.g., a surgical procedure or an engineering demonstration) and apply expert reasoning, not just basic visual perception. **Why it matters:** Most video benchmarks test surface-level perception. MMVU requires applying real domain expertise to video content, and every example is annotated by human experts from scratch with strict quality controls. **Known limitations:** As with most expert-level benchmarks, coverage is limited to the 27 selected subjects; performance may vary significantly across disciplines not represented in the dataset. ## What MMVU Measures MMVU evaluates foundation models on expert-level, multi-discipline video understanding, requiring both domain-specific knowledge and expert-level reasoning to analyze specialized videos. Its four core disciplines — Science, Healthcare, Humanities & Social Sciences, and Engineering — span 27 subjects in total. Each of the 3,000 questions is annotated from scratch by human experts and enriched with reasoning rationales and relevant domain knowledge, enabling in-depth error analysis beyond simple accuracy scores. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Video / expert-domain reasoning | | Metric | % accuracy | | Number of questions | 3,000 | | Disciplines | 4 core (27 subjects) | | Saturation | Low | | Created by | Zhao et al. (Yale NLP) | | Source paper | [MMVU (arXiv 2501.12380)](https://arxiv.org/abs/2501.12380) | ## How MMVU Is Scored Models answer expert-level questions about specialized-domain videos and are scored on % accuracy against expert-annotated ground truth. The dataset's reasoning rationales allow deeper analysis of *why* a model's answer is correct or incorrect, beyond the headline accuracy number. ## State-of-the-Art Results | Rank | Model | Score | Source | Date | |---|---|---|---|---| | 1 | [Kimi K3](/models/moonshot/kimi-k3) | 82.1% | [Kimi K3 technical report](https://www.kimi.com/blog/kimi-k3) | 2026-07 | *Score sourced from Moonshot AI's Kimi K3 announcement, July 2026.* ## MMVU on Benchgen > No Benchgen results yet — [be the first to run MMVU](/benchmarks/mmvu). ## MMVU vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **MMVU** | Expert-level, multi-discipline video understanding | 3,000 | Low | | [Video-MME](/benchmarks/video-mme) | General multimodal video understanding | 2,700 | Low | | [MMMU-Pro](/benchmarks/mmmu-pro) | Multi-discipline multimodal image understanding | — | Low | | [SciCode](/benchmarks/scicode) | Scientific coding | — | Low | ## Run MMVU on Your Model Benchgen lets you run MMVU against your own multimodal model, tracking expert-domain video reasoning accuracy across disciplines to identify where fine-tuning improves real-world domain understanding. ## Frequently Asked Questions

What is MMVU?

MMVU is an expert-level, multi-discipline video understanding benchmark with 3,000 questions across 27 subjects in Science, Healthcare, Humanities & Social Sciences, and Engineering.

What does a good score look like on MMVU?

At release, System-2-capable models like o1 and Gemini 2.0 Flash Thinking led the pack but still fell short of human expertise. By mid-2026, frontier models such as Kimi K3 report scores around 82%.

Who created MMVU?

MMVU was created by Yilun Zhao and collaborators at Yale NLP (arXiv:2501.12380, January 2025).

MMVU — Results

MMVU — Phases

MMVU