# TOMATO ![Category](https://img.shields.io/badge/Category-Multimodal-1f6feb) ![Metric](https://img.shields.io/badge/Metric-Accuracy-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-1%2C484-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-low-d29922) ![Created](https://img.shields.io/badge/Created-Oct%202024-555) [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/abs/2410.23266) [![GitHub](https://img.shields.io/badge/GitHub-Code-181717?logo=github&logoColor=white)](https://github.com/yale-nlp/TOMATO) > **Quick answer:** TOMATO (Temporal Reasoning Multimodal Evaluation) is a 1,484-question video benchmark built to prove whether multimodal foundation models actually reason across a sequence of frames, or are quietly solving questions from a single frame or out-of-order clues. ## At a Glance **What it tests:** Genuine temporal reasoning over video — action counting, direction, rotation, shape/trend change, velocity/frequency, and visual cues — using videos specifically constructed to resist single-frame or frame-order shortcuts. **Why it matters:** Prior video benchmarks overstated temporal understanding because many of their questions were solvable from a single frame; TOMATO's three-principle design (Multi-Frame Gain, Frame Order Sensitivity, Frame Information Disparity) makes shortcut-solving much harder, giving a more honest read on video reasoning. **Known limitations:** The benchmark is deliberately adversarial to shortcuts, so scores are lower across the board than on conventional video QA sets — useful for stress-testing but not directly comparable to easier video benchmarks. ## What TOMATO Measures TOMATO comprises 1,484 carefully curated, human-annotated questions spanning six tasks — action count, direction, rotation, shape & trend, velocity & frequency, and visual cues — applied across 1,417 videos. 805 of these videos were self-recorded or generated by the authors specifically to introduce counterfactual scenes, composite motions, and zoomed-in views that standard footage doesn't provide, spanning human-centric, real-world, and simulated scenarios. The paper's evaluation found a 57.3-percentage-point gap between human performance and the best-performing model, and — critically — showed that models which accurately caption individual frames still frequently fail to stitch those captions into a correct overall temporal conclusion (e.g., correctly describing each step of a rotation but concluding the wrong overall direction). Because TOMATO's videos are engineered against single-frame and reordering shortcuts, high scores on it are a stronger signal of real video-temporal reasoning than high scores on conventional video QA benchmarks. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Multimodal (video temporal reasoning) | | Metric | Accuracy (%) across 6 temporal-reasoning task types | | Number of tasks | 1,484 human-annotated questions on 1,417 videos | | Saturation | Low — ~57-point human/model gap at release | | Created by | Ziyao Shangguan, Chuhan Li, Yuxuan Ding, Yanan Zheng, Yilun Zhao, Tesca Fitzgerald, Arman Cohan | | Source paper | [Shangguan et al. 2024](https://arxiv.org/abs/2410.23266) | | Code | [GitHub](https://github.com/yale-nlp/TOMATO) | | Dataset | [Hugging Face](https://huggingface.co/datasets/yale-nlp/TOMATO) | ## How TOMATO Is Scored Each question is scored as correct/incorrect against ground truth, then aggregated into overall accuracy and per-task-type accuracy, letting teams see whether a model is weak on a specific temporal aspect (e.g., rotation vs. velocity) rather than just an overall number. ## State-of-the-Art Results > Scores vary by frame-sampling rate and model release. See the [official repository](https://github.com/yale-nlp/TOMATO) for current standings. ## TOMATO on Benchgen > No Benchgen results yet — [be the first to run TOMATO](/benchmarks/shangguan-et-al-yale-university/tomato). ## TOMATO vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **TOMATO** | Video temporal reasoning resistant to single-frame shortcuts | 1,484 | low | | [MVBench](/benchmarks/li-et-al-shanghai-ai-laboratory/mvbench) | 20-task comprehensive video understanding | — | medium | | [TempCompass](/benchmarks/liu-et-al-peking-university/tempcompass) | Temporal-perception-specific video QA with conflicting videos | — | low | TOMATO and TempCompass both target the same weak spot — genuine temporal reasoning versus single-frame shortcuts — while MVBench spans a broader 20-task video-understanding surface. ## Run TOMATO on Your Model Benchgen lets teams run the full TOMATO suite against their own model or agent stack on demand, tracking per-task-type temporal-reasoning scores over time to catch regressions that a single aggregate video-QA score would hide. ## Related Models & Benchmarks Explore related multimodal benchmarks like [MVBench](/benchmarks/li-et-al-shanghai-ai-laboratory/mvbench) and [TempCompass](/benchmarks/liu-et-al-peking-university/tempcompass) to compare broad video understanding with TOMATO's shortcut-resistant temporal-reasoning focus. ## Frequently Asked Questions

What is TOMATO?

TOMATO is a 1,484-question video benchmark designed to test genuine temporal reasoning across frames, using videos engineered to resist single-frame or frame-order shortcuts.

What does a good TOMATO score look like?

At release, even the best-performing model trailed human accuracy by 57.3 percentage points, so any model materially closing that gap represents a strong result.

Who created TOMATO?

TOMATO was created by Ziyao Shangguan and collaborators at Yale University.

Rank	Model	Score
1	qwen2-vl-72b	37.9
2	gpt-4o	37.7
3	gemini-1-5-pro	36.1
4	gemini-2-0-flash	33.7
5	qwen2-vl-7b	31.5
6	gpt-4o-mini	28.8
7	claude-3-opus	28.2
8	claude-3-5-sonnet	27.8
9	gemini-1-5-flash	27.8
10	claude-3-haiku	26.2

TOMATO — Results

TOMATO — Evaluations