Video-MME — Results
| Rank | Model | Score |
|---|---|---|
| 1 | kimi-k3 | 90 |
Video-MME
1 phaseActive
First comprehensive multimodal video-understanding benchmark — 900 videos, 254 hours, 2,700 QA pairs across 6 domains. Metric: % accuracy.
Overview
Video-MME
Quick answer: Video-MME (Fu et al., 2024) is the first full-spectrum, comprehensive evaluation benchmark of multimodal LLMs in video analysis, covering 900 videos (254 hours) across 6 primary domains and 30 subfields, with 2,700 expert-annotated question-answer pairs. Videos range from 11 seconds to 1 hour and can be evaluated with or without subtitles/audio. Kimi K3 scores 90.0% in the "with subtitles" setting as of July 2026.
At a Glance
What it tests: A model's ability to understand video content — visual, audio, and subtitle information — across short, medium, and long videos spanning 6 domains and 30 subfields.
Why it matters: Prior video benchmarks were narrow in scope or duration. Video-MME's diversity (video type, duration, modality) and rigorous manual annotation make it a broadly trusted reference point for multimodal video understanding.
Known limitations: Requires native video/audio input support — text-only or image-only models cannot be meaningfully evaluated. Scores can vary significantly depending on whether subtitles/audio are provided to the model.
What Video-MME Measures
Video-MME distinguishes itself through four features: diversity across 6 primary visual domains and 30 subfields; duration breadth spanning short (11s–2min), medium (4–15min), and long (30–60min) videos; multi-modal breadth by incorporating subtitles and audio alongside frames; and high-quality manual annotation by expert reviewers who repeatedly watched each video before writing questions. The 900 videos were manually selected and annotated to produce 2,700 question-answer pairs, and evaluations report scores both with and without subtitles to isolate the contribution of textual/audio grounding.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Video / multimodal reasoning |
| Metric | % accuracy |
| Number of QA pairs | 2,700 |
| Number of videos | 900 (254 hours total) |
| Domains | 6 primary domains, 30 subfields |
| Saturation | Low |
| Created by | Fu et al. |
| Source paper | Video-MME (arXiv 2405.21075) |
| Dataset | lmms-lab/Video-MME on HuggingFace |
How Video-MME Is Scored
Models answer multiple-choice questions about each video and are scored on % accuracy. Results are commonly reported in two settings — "without subtitles" (video frames only) and "with subtitles" (frames plus subtitle/audio transcript) — with the latter typically yielding higher scores due to the additional textual grounding.
State-of-the-Art Results
| Rank | Model | Score | Source | Date |
|---|---|---|---|---|
| 1 | Kimi K3 | 90.0% (w/ subtitles) | Kimi K3 technical report | 2026-07 |
Score sourced from Moonshot AI's Kimi K3 announcement, July 2026.
Video-MME on Benchgen
No Benchgen results yet — be the first to run Video-MME.
Video-MME vs Other Benchmarks
| Benchmark | What it tests | Tasks | Saturation |
|---|---|---|---|
| Video-MME | Multimodal video understanding | 2,700 QA pairs | Low |
| MMVU | Expert-level, multi-discipline video understanding | 3,000 | Low |
| MMMU-Pro | Multi-discipline multimodal (image) understanding | — | Low |
| ZeroBench | Extremely hard visual reasoning | 100 | Low |
Run Video-MME on Your Model
Benchgen lets you run Video-MME against your own multimodal model, tracking accuracy across video duration bands and subtitle configurations to understand where your model's video understanding breaks down.