# Video-MME (Long, No Subtitles) ![Category](https://img.shields.io/badge/Category-Multimodal-1f6feb) ![Metric](https://img.shields.io/badge/Metric-Accuracy-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-300%20videos-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-low-d29922) ![Created](https://img.shields.io/badge/Created-May%202024-555) [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/abs/2405.21075) [![GitHub](https://img.shields.io/badge/GitHub-Code-181717?logo=github&logoColor=white)](https://github.com/BradyFU/Video-MME) > **Quick answer:** Video-MME's long, no-subtitles split isolates the hardest evaluation slice of the first full-spectrum MLLM video benchmark — long-duration videos (up to 1 hour) evaluated using only visual frames, with subtitle text deliberately withheld to test genuine video comprehension rather than reading capability. ## At a Glance **What it tests:** Long-form video comprehension (videos up to an hour long) using purely visual understanding, without the subtitle-text crutch that can let a model "cheat" by reading dialogue instead of watching the video. **Why it matters:** Video-MME's full design spans short/medium/long durations and optionally includes subtitles and audio; the long-duration, no-subtitles configuration specifically isolates whether a model can sustain contextual understanding over long visual sequences on its own, which is the hardest and most video-comprehension-pure setting the benchmark offers. **Known limitations:** Because this is a specific evaluation configuration of the broader Video-MME dataset rather than a separately collected dataset, results must be compared against the same configuration (long duration, no subtitles) to be meaningful — mixing in short/medium durations or subtitle-assisted results would not be a fair comparison. ## What Video-MME (Long, No Subtitles) Measures Video-MME is the first full-spectrum, multi-modal evaluation benchmark of MLLMs in video analysis, distinguished by diversity in video types (6 primary visual domains, 30 subfields), duration (short/medium/long, from 11 seconds to 1 hour), breadth of modalities (video frames, subtitles, and audio), and quality of annotation (900 videos manually selected and annotated by expert annotators, yielding 2,700 question-answer pairs). The long-duration subset specifically covers the hour-scale videos within this collection. The no-subtitles evaluation configuration deliberately withholds subtitle text from the model, forcing it to answer purely from visual (and optionally audio) understanding of the long video rather than relying on reading dialogue or on-screen text as a shortcut. This isolates the genuinely hardest capability Video-MME probes: sustaining coherent contextual understanding across an hour of visual content without any textual crutch. The paper's evaluation found Gemini 1.5 Pro to be the best-performing commercial model overall, significantly outperforming open-source alternatives, with performance dropping notably as video duration increases and subtitle assistance is removed — directly motivating continued research into longer-context, more robust video comprehension. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Multimodal (long-duration video comprehension, no subtitles) | | Metric | Accuracy (%) on multiple-choice question-answer pairs | | Number of tasks | ~300 long-duration videos (of Video-MME's 900 total, evenly split short/medium/long) | | Saturation | Low — this is the hardest configuration of Video-MME, with scores dropping notably vs. shorter/subtitle-assisted settings | | Created by | Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, and collaborators | | Source paper | [Fu et al. 2024](https://arxiv.org/abs/2405.21075) | | Code | [GitHub](https://github.com/BradyFU/Video-MME) | | Dataset | [Hugging Face](https://huggingface.co/datasets/lmms-lab/Video-MME) | ## How Video-MME (Long, No Subtitles) Is Scored Questions are scored as multiple-choice accuracy against ground truth, restricted to the long-duration video subset and evaluated with subtitle input withheld, then reported alongside the benchmark's other duration/modality configurations (short/medium/long, with/without subtitles) so teams can see exactly how performance degrades as duration increases and textual assistance is removed. ## State-of-the-Art Results > Scores vary substantially between subtitle-assisted and no-subtitles settings, and decline further as duration increases. See the [official leaderboard](https://video-mme.github.io/) for current standings broken out by duration and modality configuration. ## Video-MME (Long, No Subtitles) on Benchgen > No Benchgen results yet — [be the first to run Video-MME (Long, No Subtitles)](/benchmarks/fu-et-al-xiamen-university/video-mme-long-no-subtitles). ## Video-MME (Long, No Subtitles) vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **Video-MME (Long, No Subtitles)** | Hour-long video comprehension, no subtitle assistance | ~300 | low | | [MMBench-Video](/benchmarks/fang-et-al-shanghai-ai-laboratory/mmbench-video) | Long-form, multi-shot video understanding | 1,998 | low | | [Video-MMMU](/benchmarks/hu-et-al-nanyang-technological-university/video-mmmu) | Knowledge acquisition from lecture videos | 900 | low | This configuration represents the hardest slice of Video-MME's duration/modality matrix, complementing MMBench-Video's free-form long-video QA and Video-MMMU's knowledge-acquisition focus. ## Run Video-MME (Long, No Subtitles) on Your Model Benchgen lets teams run the long-duration, no-subtitles configuration of Video-MME against their own model or agent versions on demand, tracking accuracy over time to catch regressions specifically in sustained, subtitle-free long-video comprehension. ## Related Models & Benchmarks Explore related multimodal video benchmarks like [MMBench-Video](/benchmarks/fang-et-al-shanghai-ai-laboratory/mmbench-video) and [Video-MMMU](/benchmarks/hu-et-al-nanyang-technological-university/video-mmmu) to compare long-form comprehension with knowledge-acquisition-focused video evaluation. ## Frequently Asked Questions

What is Video-MME (Long, No Subtitles)?

It's the hardest evaluation configuration of Video-MME — the first full-spectrum MLLM video benchmark — isolating hour-long videos evaluated without subtitle assistance to test pure visual video comprehension.

What does a good score look like on this configuration?

Scores in this configuration are meaningfully lower than on shorter durations or subtitle-assisted settings; Gemini 1.5 Pro was the strongest commercial model in the original evaluation, but all models showed degraded performance as duration increased and subtitles were removed.

Who created Video-MME?

Video-MME was created by Chaoyou Fu and collaborators at Xiamen University and Tencent Youtu Lab.

Rank	Model	Score
1	gemini-1-5-pro	67.4
2	gpt-4o	65.3
3	qwen2-vl-72b	62.2
4	gemini-1-5-flash	61.1
5	llava-onevision-72b	60
6	gpt-4o-mini	58.6
7	gpt-4v	53.5
8	minicpm-v-2-6	51.8
9	claude-3-5-sonnet	51.2
10	qwen-vl-max	48.9
11	internvl-chat-v1-5	45.6

Video-MME (Long, No Subtitles) — Results

Video-MME (Long, No Subtitles) — Evaluations