# Video-MME ![Category](https://img.shields.io/badge/Category-Video%20Understanding-1f6feb) ![Metric](https://img.shields.io/badge/Metric-%25%20Accuracy-2da44e) ![QA%20Pairs](https://img.shields.io/badge/QA%20Pairs-2%2C700-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-Low-d29922) ![Created](https://img.shields.io/badge/Created-May%202024-555) [![Paper](https://img.shields.io/badge/Paper-arXiv%202405.21075-b31b1b)](https://arxiv.org/abs/2405.21075) [![Dataset](https://img.shields.io/badge/Dataset-HuggingFace-ff9d00?logo=huggingface&logoColor=white)](https://huggingface.co/datasets/lmms-lab/Video-MME) > **Quick answer:** Video-MME (Fu et al., 2024) is the first full-spectrum, comprehensive evaluation benchmark of multimodal LLMs in video analysis, covering 900 videos (254 hours) across 6 primary domains and 30 subfields, with 2,700 expert-annotated question-answer pairs. Videos range from 11 seconds to 1 hour and can be evaluated with or without subtitles/audio. Kimi K3 scores 90.0% in the "with subtitles" setting as of July 2026. ## At a Glance **What it tests:** A model's ability to understand video content — visual, audio, and subtitle information — across short, medium, and long videos spanning 6 domains and 30 subfields. **Why it matters:** Prior video benchmarks were narrow in scope or duration. Video-MME's diversity (video type, duration, modality) and rigorous manual annotation make it a broadly trusted reference point for multimodal video understanding. **Known limitations:** Requires native video/audio input support — text-only or image-only models cannot be meaningfully evaluated. Scores can vary significantly depending on whether subtitles/audio are provided to the model. ## What Video-MME Measures Video-MME distinguishes itself through four features: diversity across 6 primary visual domains and 30 subfields; duration breadth spanning short (11s–2min), medium (4–15min), and long (30–60min) videos; multi-modal breadth by incorporating subtitles and audio alongside frames; and high-quality manual annotation by expert reviewers who repeatedly watched each video before writing questions. The 900 videos were manually selected and annotated to produce 2,700 question-answer pairs, and evaluations report scores both with and without subtitles to isolate the contribution of textual/audio grounding. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Video / multimodal reasoning | | Metric | % accuracy | | Number of QA pairs | 2,700 | | Number of videos | 900 (254 hours total) | | Domains | 6 primary domains, 30 subfields | | Saturation | Low | | Created by | Fu et al. | | Source paper | [Video-MME (arXiv 2405.21075)](https://arxiv.org/abs/2405.21075) | | Dataset | [lmms-lab/Video-MME on HuggingFace](https://huggingface.co/datasets/lmms-lab/Video-MME) | ## How Video-MME Is Scored Models answer multiple-choice questions about each video and are scored on % accuracy. Results are commonly reported in two settings — "without subtitles" (video frames only) and "with subtitles" (frames plus subtitle/audio transcript) — with the latter typically yielding higher scores due to the additional textual grounding. ## State-of-the-Art Results | Rank | Model | Score | Source | Date | |---|---|---|---|---| | 1 | [Kimi K3](/models/moonshot/kimi-k3) | 90.0% (w/ subtitles) | [Kimi K3 technical report](https://www.kimi.com/blog/kimi-k3) | 2026-07 | *Score sourced from Moonshot AI's Kimi K3 announcement, July 2026.* ## Video-MME on Benchgen > No Benchgen results yet — [be the first to run Video-MME](/benchmarks/video-mme). ## Video-MME vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **Video-MME** | Multimodal video understanding | 2,700 QA pairs | Low | | [MMVU](/benchmarks/mmvu) | Expert-level, multi-discipline video understanding | 3,000 | Low | | [MMMU-Pro](/benchmarks/mmmu-pro) | Multi-discipline multimodal (image) understanding | — | Low | | [ZeroBench](/benchmarks/zerobench) | Extremely hard visual reasoning | 100 | Low | ## Run Video-MME on Your Model Benchgen lets you run Video-MME against your own multimodal model, tracking accuracy across video duration bands and subtitle configurations to understand where your model's video understanding breaks down. ## Frequently Asked Questions

What is Video-MME?

Video-MME is the first comprehensive, full-spectrum benchmark for evaluating multimodal LLMs on video analysis, covering 900 videos and 2,700 expert-annotated question-answer pairs across 6 domains and 30 subfields.

What does a good score look like on Video-MME?

At release (2024), Gemini 1.5 Pro led as the best commercial model, significantly outperforming open-source models. By mid-2026, frontier multimodal models such as Kimi K3 report scores around 90% in the with-subtitles setting.

Who created Video-MME?

Video-MME was created by Chaoyou Fu and collaborators (arXiv:2405.21075, 2024), with a project page at video-mme.github.io.

Video-MME — Results

Video-MME — Evaluations