TempCompass
1 phaseActive
Conflicting-video benchmark spanning multiple temporal aspects and 4 task formats (multi-choice, yes/no, caption matching, captioning).
Overview
TempCompass
Quick answer: TempCompass evaluates whether Video LLMs genuinely understand temporal aspects of video — like speed, direction, and event order — by pairing each video with a "conflicting" counterpart that shares the same static content but differs only in the temporal dimension being tested.
At a Glance
What it tests: Multiple distinct temporal aspects (action, direction, speed, order, attribute change) across four task formats — multi-choice QA, yes/no QA, caption matching, and caption generation.
Why it matters: Most prior video benchmarks used only multi-choice QA and couldn't isolate specific temporal aspects; TempCompass's conflicting-video design and multi-format questions prevent models from exploiting single-frame bias or language priors.
Known limitations: Caption-generation scoring partly relies on ChatGPT-based automatic evaluation, introducing some judge-model dependency alongside the rule-based components.
What TempCompass Measures
TempCompass constructs pairs of "conflicting" videos that share identical static content but differ in one specific temporal aspect (e.g., the same objects moving in opposite directions), which prevents a model from answering correctly using only single-frame appearance or generic language priors. Questions are generated via a two-step pipeline: humans first annotate meta-information about each video, then an LLM generates the natural-language instruction from that annotation.
The benchmark evaluates across five temporal aspects — action, direction, speed, order, and attribute change — using four distinct task formats, letting researchers see whether a model's temporal weakness is format-specific (e.g., good at multi-choice but poor at open captioning) rather than relying on one question style.
At release, the paper's evaluation of 8 state-of-the-art Video LLMs and 3 Image LLMs found consistently poor temporal perception across the board, motivating continued benchmark tracking as new Video LLMs are released.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Multimodal (video temporal perception) |
| Metric | Accuracy (%), varies by task format (multi-choice, yes/no, caption matching, captioning) |
| Number of tasks | 410 processed videos across 5 temporal aspects and 4 task formats |
| Saturation | Low — leading Video LLMs still show poor temporal perception |
| Created by | Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, Lu Hou |
| Source paper | Liu et al. 2024 |
| Code | GitHub |
| Dataset | Hugging Face |
How TempCompass Is Scored
Multi-choice QA, yes/no QA, and caption matching are scored via rule-based exact/pattern matching against ground truth; caption generation is scored with an LLM-based evaluator (ChatGPT) that checks whether the generated caption correctly reflects the tested temporal aspect. Scores are reported per temporal aspect and per task format, then averaged into an overall accuracy.
State-of-the-Art Results
Scores vary by model release and frame-sampling settings. See the official leaderboard for current standings.
TempCompass on Benchgen
No Benchgen results yet — be the first to run TempCompass.
TempCompass vs Other Benchmarks
| Benchmark | What it tests | Tasks | Saturation |
|---|---|---|---|
| TempCompass | Temporal aspects via conflicting-video pairs, 4 task formats | 410 videos | low |
| TOMATO | Temporal reasoning resistant to single-frame/order shortcuts | 1,484 | low |
| MVBench | 20-task comprehensive video understanding | — | medium |
TempCompass's conflicting-video pairs and multi-format questions make it a finer-grained diagnostic than MVBench's broader task coverage, while TOMATO applies a similar shortcut-resistance philosophy with a different question-generation approach.
Run TempCompass on Your Model
Benchgen lets teams run TempCompass against their own model or agent versions on demand, breaking results out by temporal aspect and task format to catch regressions that a single video-QA accuracy number would hide.
Related Models & Benchmarks
Explore related multimodal benchmarks like TOMATO and MVBench to compare different approaches to video temporal-reasoning evaluation.