Perception Test
1 phaseActive
Google DeepMind video perception benchmark covering multiple-choice QA, grounding, tracking, and action/sound localization across diverse everyday videos.
Overview
Perception Test
Quick answer: Perception Test is a Google DeepMind benchmark built from ~11,600 real-world videos, testing multimodal models across four skill areas (memory, abstraction, physics, semantics) via multiple-choice video QA, object/point tracking, temporal action localization, and audio localization tasks.
At a Glance
What it tests: Broad multimodal video perception — visual, audio, and text reasoning about everyday scenes across memory, abstraction, physics, and semantics skill categories.
Why it matters: Unlike single-task video datasets, Perception Test was purpose-built (not repurposed from existing captioning/QA corpora) to jointly evaluate multiple task types on the same set of crowd-sourced videos, reducing task-specific overfitting.
Known limitations: As a NeurIPS 2023 Datasets & Benchmarks paper aimed at general perception diagnostics, its videos are shorter/simpler than benchmarks targeting long-form or highly specialized video understanding.
What Perception Test Measures
Perception Test evaluates models on ~11,600 short, real-world videos filmed by crowd-sourced participants performing everyday activities and games, annotated across four task types: multiple-choice video QA, object and point tracking, temporal action and sound localization, and grounded video QA. Each video is further tagged along four cognitive skill dimensions — memory, abstraction, physics, and semantics — so results can be broken down by both task type and underlying reasoning skill.
The benchmark was explicitly designed as a general-purpose diagnostic tool rather than a task-specific leaderboard: its authors curated fresh video content (rather than reusing existing video datasets) specifically to test multimodal perception in a controlled, skill-labeled way.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Multimodal (video QA, tracking, temporal/audio localization) |
| Metric | Multiple-choice accuracy (%) for video QA; IoU/mAP-based metrics for tracking and localization tasks |
| Number of tasks | ~11,600 real-world videos annotated across QA, tracking, and localization task types |
| Saturation | Medium — frontier video-LLMs perform well on QA but still show gaps on temporal localization and tracking |
| Created by | Viorica Pătrăucean and 22 co-authors (Google DeepMind) |
| Source paper | Pătrăucean et al. 2023 (NeurIPS 2023 D&B) |
| Code & data | GitHub |
How Perception Test Is Scored
Multiple-choice video QA is scored by accuracy against the correct answer option. Object/point tracking is scored using standard tracking metrics (e.g., IoU-based success rate), and temporal action/sound localization tasks are scored with mean Average Precision (mAP) over predicted time intervals. Aggregate leaderboards typically report per-task-type scores broken down further by the four skill categories.
State-of-the-Art Results
Scores fluctuate by model release cadence and evaluation harness (prompt template, sampling rate, frame count). See the official repository for current standings across QA, tracking, and localization tasks.
Perception Test on Bengen
No Benchgen results yet — be the first to run Perception Test.
Perception Test vs Other Benchmarks
| Benchmark | What it tests | Tasks | Saturation |
|---|---|---|---|
| Perception Test | Video QA + tracking + localization across 4 skill areas | 11,600 videos | medium |
| MVBench | Multi-choice video QA across temporal reasoning tasks | — | medium |
| VideoMMMU | College-level knowledge acquisition from educational videos | — | low |
Perception Test's broad skill-labeled video suite complements narrower long-form or knowledge-focused video benchmarks like MVBench and VideoMMMU.
Run Perception Test on Your Model
Benchgen lets teams run the full Perception Test suite (QA, tracking, localization) against their own model versions on demand, with results broken down by both task type and cognitive skill category.
Related Models & Benchmarks
Explore related multimodal video benchmarks like MVBench and VideoMMMU to compare short-form perception and long-form knowledge evaluation approaches.