TOMATO — Results
| Rank | Model | Score |
|---|---|---|
| 1 | qwen2-vl-72b | 37.9 |
| 2 | gpt-4o | 37.7 |
| 3 | gemini-1-5-pro | 36.1 |
| 4 | gemini-2-0-flash | 33.7 |
| 5 | qwen2-vl-7b | 31.5 |
| 6 | gpt-4o-mini | 28.8 |
| 7 | claude-3-opus | 28.2 |
| 8 | claude-3-5-sonnet | 27.8 |
| 9 | gemini-1-5-flash | 27.8 |
| 10 | claude-3-haiku | 26.2 |
TOMATO
1 phaseActive
1,484 human-annotated questions across 6 temporal-reasoning tasks on 1,417 videos, designed to prevent single-frame shortcut solving.
Overview
TOMATO
Quick answer: TOMATO (Temporal Reasoning Multimodal Evaluation) is a 1,484-question video benchmark built to prove whether multimodal foundation models actually reason across a sequence of frames, or are quietly solving questions from a single frame or out-of-order clues.
At a Glance
What it tests: Genuine temporal reasoning over video — action counting, direction, rotation, shape/trend change, velocity/frequency, and visual cues — using videos specifically constructed to resist single-frame or frame-order shortcuts.
Why it matters: Prior video benchmarks overstated temporal understanding because many of their questions were solvable from a single frame; TOMATO's three-principle design (Multi-Frame Gain, Frame Order Sensitivity, Frame Information Disparity) makes shortcut-solving much harder, giving a more honest read on video reasoning.
Known limitations: The benchmark is deliberately adversarial to shortcuts, so scores are lower across the board than on conventional video QA sets — useful for stress-testing but not directly comparable to easier video benchmarks.
What TOMATO Measures
TOMATO comprises 1,484 carefully curated, human-annotated questions spanning six tasks — action count, direction, rotation, shape & trend, velocity & frequency, and visual cues — applied across 1,417 videos. 805 of these videos were self-recorded or generated by the authors specifically to introduce counterfactual scenes, composite motions, and zoomed-in views that standard footage doesn't provide, spanning human-centric, real-world, and simulated scenarios.
The paper's evaluation found a 57.3-percentage-point gap between human performance and the best-performing model, and — critically — showed that models which accurately caption individual frames still frequently fail to stitch those captions into a correct overall temporal conclusion (e.g., correctly describing each step of a rotation but concluding the wrong overall direction).
Because TOMATO's videos are engineered against single-frame and reordering shortcuts, high scores on it are a stronger signal of real video-temporal reasoning than high scores on conventional video QA benchmarks.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Multimodal (video temporal reasoning) |
| Metric | Accuracy (%) across 6 temporal-reasoning task types |
| Number of tasks | 1,484 human-annotated questions on 1,417 videos |
| Saturation | Low — ~57-point human/model gap at release |
| Created by | Ziyao Shangguan, Chuhan Li, Yuxuan Ding, Yanan Zheng, Yilun Zhao, Tesca Fitzgerald, Arman Cohan |
| Source paper | Shangguan et al. 2024 |
| Code | GitHub |
| Dataset | Hugging Face |
How TOMATO Is Scored
Each question is scored as correct/incorrect against ground truth, then aggregated into overall accuracy and per-task-type accuracy, letting teams see whether a model is weak on a specific temporal aspect (e.g., rotation vs. velocity) rather than just an overall number.
State-of-the-Art Results
Scores vary by frame-sampling rate and model release. See the official repository for current standings.
TOMATO on Benchgen
No Benchgen results yet — be the first to run TOMATO.
TOMATO vs Other Benchmarks
| Benchmark | What it tests | Tasks | Saturation |
|---|---|---|---|
| TOMATO | Video temporal reasoning resistant to single-frame shortcuts | 1,484 | low |
| MVBench | 20-task comprehensive video understanding | — | medium |
| TempCompass | Temporal-perception-specific video QA with conflicting videos | — | low |
TOMATO and TempCompass both target the same weak spot — genuine temporal reasoning versus single-frame shortcuts — while MVBench spans a broader 20-task video-understanding surface.
Run TOMATO on Your Model
Benchgen lets teams run the full TOMATO suite against their own model or agent stack on demand, tracking per-task-type temporal-reasoning scores over time to catch regressions that a single aggregate video-QA score would hide.
Related Models & Benchmarks
Explore related multimodal benchmarks like MVBench and TempCompass to compare broad video understanding with TOMATO's shortcut-resistant temporal-reasoning focus.