# LVBench ![Category](https://img.shields.io/badge/Category-Multimodal-1f6feb) ![Metric](https://img.shields.io/badge/Metric-%25%20Accuracy-2da44e) ![Categories](https://img.shields.io/badge/Categories-6%20major%2F21%20sub-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-Low-d29922) ![Created](https://img.shields.io/badge/Created-Jun%202024-555) [![Paper](https://img.shields.io/badge/Paper-arXiv%202406.08035-b31b1b)](https://arxiv.org/abs/2406.08035) [![GitHub](https://img.shields.io/badge/GitHub-THUDM%2FLVBench-181717?logo=github&logoColor=white)](https://github.com/THUDM/LVBench) [![Dataset](https://img.shields.io/badge/Dataset-HuggingFace-ff9d00?logo=huggingface&logoColor=white)](https://huggingface.co/datasets/THUDM/LVBench) > **Quick answer:** LVBench (Wang et al., Tsinghua University / Zhipu AI, 2024) is a benchmark built specifically to test extreme long-video understanding — publicly sourced videos spanning TV series, sports broadcasts, and surveillance footage, organized into 6 major categories and 21 subcategories, designed to challenge multimodal models' long-term memory and extended comprehension. Qwen3.8-Flash-Next scores 76.6% as of August 2026. ## At a Glance **What it tests:** Whether a multimodal LLM can comprehend and extract information from videos spanning many minutes to multiple hours, rather than the under-one-minute clips most prior video benchmarks used. **Why it matters:** Real-world video-understanding use cases — embodied long-term decision-making, in-depth movie/show discussion, live sports commentary — require models to track information across hours of content, a capability short-video benchmarks cannot measure. **Known limitations:** The official leaderboard shows even strong 2025-era models (Deep Video Discovery at 74.2%, Seed1.5-VL-Thinking at 64.6%) still well short of saturation, and per-question-type breakdowns (ER/EU/KIR/TG/Rea/Sum) show uneven performance across reasoning types. ## What LVBench Measures LVBench addresses a gap in multimodal LLM evaluation: existing datasets focus on short videos (typically under a minute), leaving long-term memory and extended comprehension largely untested. LVBench's dataset comprises publicly sourced long videos — including TV series, sports broadcasts, and everyday surveillance footage — spanning 6 major categories and 21 subcategories, combining manual annotation with model-assisted techniques to build a diverse set of long-video comprehension and information-extraction tasks. Reported sub-metrics on the official leaderboard include Entity Recognition (ER), Event Understanding (EU), Key Information Retrieval (KIR), Temporal Grounding (TG), Reasoning (Rea), and Summarization (Sum), rolled up into an overall accuracy score. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Multimodal (extreme long-video understanding) | | Metric | % accuracy (overall, plus ER/EU/KIR/TG/Rea/Sum sub-scores) | | Video categories | 6 major categories, 21 subcategories | | Saturation | Low | | Created by | Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Xiaotao Gu, Shiyu Huang, Bin Xu, Yuxiao Dong, Ming Ding, Jie Tang (Tsinghua University / Zhipu AI) | | Source paper | [Wang et al. 2024](https://arxiv.org/abs/2406.08035) | | GitHub | [THUDM/LVBench](https://github.com/THUDM/LVBench) | | Dataset | [THUDM/LVBench on HuggingFace](https://huggingface.co/datasets/THUDM/LVBench) | ## How LVBench Is Scored Models are scored on overall % accuracy across the question set, with the official leaderboard additionally breaking results down by six question-type sub-scores (Entity Recognition, Event Understanding, Key Information Retrieval, Temporal Grounding, Reasoning, Summarization) to show where a model's long-video comprehension is weakest. ## State-of-the-Art Results | Rank | Model | Score | Source | Date | |---|---|---|---|---| | 1 | [Qwen3.8-Flash-Next](/models/qwen/qwen3-8-flash-next) | 76.6% | Qwen3.8-Flash-Next model card | 2026-08 | *Score above is self-reported by Alibaba's Qwen Team on the official Hugging Face model card and shown for context. It is not a Benchgen measurement, and is not drawn from the official THUDM/LVBench leaderboard — only this 1 score is currently tracked locally.* ## LVBench on Benchgen > No Benchgen results yet — [be the first to run LVBench](/benchmarks/tsinghua-university-zhipu-ai/lvbench). ## LVBench vs Other Benchmarks | Benchmark | What it tests | Saturation | |---|---|---| | **LVBench** | Extreme long-video understanding (hours-long videos) | Low | | [Video-MME](/benchmarks/video-mme/video-mme) | Full-spectrum video understanding, 11s-1hr videos | Low | | [MMBench-Video](/benchmarks/xinyu-fang-kangrui-mao-haodong-duan-xiangyu-zhao-yining-li-dahua-lin-kai-chen-shanghai-ai-laboratory/mmbench-video) | Long-form, multi-shot video understanding, GPT-4 judged | Low | LVBench pushes video duration further than Video-MME or MMBench-Video, specifically targeting multi-hour content (TV series, sports broadcasts, surveillance) rather than clips under an hour. ## Run LVBench on Your Model Benchgen lets you run LVBench against your own multimodal model, tracking long-video comprehension accuracy across entity recognition, event understanding, temporal grounding, and summarization sub-tasks. ## Related Models & Benchmarks - Top performers: [Qwen3.8-Flash-Next](/models/qwen/qwen3-8-flash-next) - Related benchmarks: [Video-MME](/benchmarks/video-mme/video-mme) · [MMBench-Video](/benchmarks/xinyu-fang-kangrui-mao-haodong-duan-xiangyu-zhao-yining-li-dahua-lin-kai-chen-shanghai-ai-laboratory/mmbench-video) ## Frequently Asked Questions

What is LVBench?

LVBench is a Tsinghua University / Zhipu AI benchmark (Wang et al., 2024) designed to test extreme long-video understanding using publicly sourced videos — TV series, sports broadcasts, and surveillance footage — across 6 major categories and 21 subcategories.

What does a good LVBench score look like?

Qwen3.8-Flash-Next reports 76.6% as of August 2026. On the official THUDM leaderboard, top 2025-era models range from the mid-60s to mid-70s percent, with most models well below that — the benchmark remains unsaturated.

Who created LVBench?

LVBench was created by Weihan Wang and collaborators at Tsinghua University and Zhipu AI; see the [original paper](https://arxiv.org/abs/2406.08035).

Is LVBench saturated?

No — LVBench remains a low-saturation benchmark; even leading 2025-era multimodal models on the official leaderboard score well below 80%, reflecting the difficulty of multi-hour video comprehension.

How does LVBench differ from Video-MME?

Video-MME covers videos from 11 seconds to 1 hour across 6 domains; LVBench specifically targets much longer, multi-hour videos (TV series, sports, surveillance) to test long-term memory and extended comprehension beyond what Video-MME's duration range captures.

--- *Benchmark definition paraphrased from [Wang et al. 2024](https://arxiv.org/abs/2406.08035). State-of-the-art score sourced from a published model card and attributed inline. Last updated 2026-10-31.*

LVBench — Results

LVBench — Evaluations