# TempCompass ![Category](https://img.shields.io/badge/Category-Multimodal-1f6feb) ![Metric](https://img.shields.io/badge/Metric-Accuracy-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-410%20videos-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-low-d29922) ![Created](https://img.shields.io/badge/Created-Mar%202024-555) [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/abs/2403.00476) [![GitHub](https://img.shields.io/badge/GitHub-Code-181717?logo=github&logoColor=white)](https://github.com/llyx97/TempCompass) > **Quick answer:** TempCompass evaluates whether Video LLMs genuinely understand temporal aspects of video — like speed, direction, and event order — by pairing each video with a "conflicting" counterpart that shares the same static content but differs only in the temporal dimension being tested. ## At a Glance **What it tests:** Multiple distinct temporal aspects (action, direction, speed, order, attribute change) across four task formats — multi-choice QA, yes/no QA, caption matching, and caption generation. **Why it matters:** Most prior video benchmarks used only multi-choice QA and couldn't isolate specific temporal aspects; TempCompass's conflicting-video design and multi-format questions prevent models from exploiting single-frame bias or language priors. **Known limitations:** Caption-generation scoring partly relies on ChatGPT-based automatic evaluation, introducing some judge-model dependency alongside the rule-based components. ## What TempCompass Measures TempCompass constructs pairs of "conflicting" videos that share identical static content but differ in one specific temporal aspect (e.g., the same objects moving in opposite directions), which prevents a model from answering correctly using only single-frame appearance or generic language priors. Questions are generated via a two-step pipeline: humans first annotate meta-information about each video, then an LLM generates the natural-language instruction from that annotation. The benchmark evaluates across five temporal aspects — action, direction, speed, order, and attribute change — using four distinct task formats, letting researchers see whether a model's temporal weakness is format-specific (e.g., good at multi-choice but poor at open captioning) rather than relying on one question style. At release, the paper's evaluation of 8 state-of-the-art Video LLMs and 3 Image LLMs found consistently poor temporal perception across the board, motivating continued benchmark tracking as new Video LLMs are released. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Multimodal (video temporal perception) | | Metric | Accuracy (%), varies by task format (multi-choice, yes/no, caption matching, captioning) | | Number of tasks | 410 processed videos across 5 temporal aspects and 4 task formats | | Saturation | Low — leading Video LLMs still show poor temporal perception | | Created by | Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, Lu Hou | | Source paper | [Liu et al. 2024](https://arxiv.org/abs/2403.00476) | | Code | [GitHub](https://github.com/llyx97/TempCompass) | | Dataset | [Hugging Face](https://huggingface.co/datasets/lmms-lab/TempCompass) | ## How TempCompass Is Scored Multi-choice QA, yes/no QA, and caption matching are scored via rule-based exact/pattern matching against ground truth; caption generation is scored with an LLM-based evaluator (ChatGPT) that checks whether the generated caption correctly reflects the tested temporal aspect. Scores are reported per temporal aspect and per task format, then averaged into an overall accuracy. ## State-of-the-Art Results > Scores vary by model release and frame-sampling settings. See the [official leaderboard](https://huggingface.co/spaces/lyx97/TempCompass) for current standings. ## TempCompass on Benchgen > No Benchgen results yet — [be the first to run TempCompass](/benchmarks/liu-et-al-peking-university/tempcompass). ## TempCompass vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **TempCompass** | Temporal aspects via conflicting-video pairs, 4 task formats | 410 videos | low | | [TOMATO](/benchmarks/shangguan-et-al-yale-university/tomato) | Temporal reasoning resistant to single-frame/order shortcuts | 1,484 | low | | [MVBench](/benchmarks/li-et-al-shanghai-ai-laboratory/mvbench) | 20-task comprehensive video understanding | — | medium | TempCompass's conflicting-video pairs and multi-format questions make it a finer-grained diagnostic than MVBench's broader task coverage, while TOMATO applies a similar shortcut-resistance philosophy with a different question-generation approach. ## Run TempCompass on Your Model Benchgen lets teams run TempCompass against their own model or agent versions on demand, breaking results out by temporal aspect and task format to catch regressions that a single video-QA accuracy number would hide. ## Related Models & Benchmarks Explore related multimodal benchmarks like [TOMATO](/benchmarks/shangguan-et-al-yale-university/tomato) and [MVBench](/benchmarks/li-et-al-shanghai-ai-laboratory/mvbench) to compare different approaches to video temporal-reasoning evaluation. ## Frequently Asked Questions

What is TempCompass?

TempCompass is a video benchmark that pairs videos sharing the same static content but differing in one temporal aspect, testing whether Video LLMs genuinely understand speed, direction, order, and other temporal properties across four task formats.

What does a good TempCompass score look like?

At release, even leading Video LLMs showed notably poor temporal perception across most aspects, so scores that hold up consistently across all five temporal aspects (not just one) represent genuinely strong performance.

Who created TempCompass?

TempCompass was created by Yuanxin Liu and collaborators at Peking University, accepted at ACL 2024 Findings.

TempCompass — Leaderboard

TempCompass Leaderboard