# Perception Test ![Category](https://img.shields.io/badge/Category-Multimodal-1f6feb) ![Metric](https://img.shields.io/badge/Metric-Accuracy%20%2F%20IoU-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-11%2C600%20videos-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-medium-d29922) ![Created](https://img.shields.io/badge/Created-May%202023-555) [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/abs/2305.13786) [![GitHub](https://img.shields.io/badge/GitHub-Repo-181717?logo=github&logoColor=white)](https://github.com/google-deepmind/perception_test) > **Quick answer:** Perception Test is a Google DeepMind benchmark built from ~11,600 real-world videos, testing multimodal models across four skill areas (memory, abstraction, physics, semantics) via multiple-choice video QA, object/point tracking, temporal action localization, and audio localization tasks. ## At a Glance **What it tests:** Broad multimodal video perception — visual, audio, and text reasoning about everyday scenes across memory, abstraction, physics, and semantics skill categories. **Why it matters:** Unlike single-task video datasets, Perception Test was purpose-built (not repurposed from existing captioning/QA corpora) to jointly evaluate multiple task types on the same set of crowd-sourced videos, reducing task-specific overfitting. **Known limitations:** As a NeurIPS 2023 Datasets & Benchmarks paper aimed at general perception diagnostics, its videos are shorter/simpler than benchmarks targeting long-form or highly specialized video understanding. ## What Perception Test Measures Perception Test evaluates models on ~11,600 short, real-world videos filmed by crowd-sourced participants performing everyday activities and games, annotated across four task types: **multiple-choice video QA**, **object and point tracking**, **temporal action and sound localization**, and **grounded video QA**. Each video is further tagged along four cognitive skill dimensions — memory, abstraction, physics, and semantics — so results can be broken down by both task type and underlying reasoning skill. The benchmark was explicitly designed as a general-purpose diagnostic tool rather than a task-specific leaderboard: its authors curated fresh video content (rather than reusing existing video datasets) specifically to test multimodal perception in a controlled, skill-labeled way. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Multimodal (video QA, tracking, temporal/audio localization) | | Metric | Multiple-choice accuracy (%) for video QA; IoU/mAP-based metrics for tracking and localization tasks | | Number of tasks | ~11,600 real-world videos annotated across QA, tracking, and localization task types | | Saturation | Medium — frontier video-LLMs perform well on QA but still show gaps on temporal localization and tracking | | Created by | Viorica Pătrăucean and 22 co-authors (Google DeepMind) | | Source paper | [Pătrăucean et al. 2023](https://arxiv.org/abs/2305.13786) (NeurIPS 2023 D&B) | | Code & data | [GitHub](https://github.com/google-deepmind/perception_test) | ## How Perception Test Is Scored Multiple-choice video QA is scored by accuracy against the correct answer option. Object/point tracking is scored using standard tracking metrics (e.g., IoU-based success rate), and temporal action/sound localization tasks are scored with mean Average Precision (mAP) over predicted time intervals. Aggregate leaderboards typically report per-task-type scores broken down further by the four skill categories. ## State-of-the-Art Results > Scores fluctuate by model release cadence and evaluation harness (prompt template, sampling rate, frame count). See the [official repository](https://github.com/google-deepmind/perception_test) for current standings across QA, tracking, and localization tasks. ## Perception Test on Bengen > No Benchgen results yet — [be the first to run Perception Test](/benchmarks/viorica-patraucean-et-al-google-deepmind/perceptiontest). ## Perception Test vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **Perception Test** | Video QA + tracking + localization across 4 skill areas | 11,600 videos | medium | | [MVBench](/benchmarks/mvbench) | Multi-choice video QA across temporal reasoning tasks | — | medium | | [VideoMMMU](/benchmarks/videommmu) | College-level knowledge acquisition from educational videos | — | low | Perception Test's broad skill-labeled video suite complements narrower long-form or knowledge-focused video benchmarks like MVBench and VideoMMMU. ## Run Perception Test on Your Model Benchgen lets teams run the full Perception Test suite (QA, tracking, localization) against their own model versions on demand, with results broken down by both task type and cognitive skill category. ## Related Models & Benchmarks Explore related multimodal video benchmarks like [MVBench](/benchmarks/mvbench) and [VideoMMMU](/benchmarks/videommmu) to compare short-form perception and long-form knowledge evaluation approaches. ## Frequently Asked Questions

What is Perception Test?

Perception Test is a Google DeepMind benchmark of ~11,600 real-world videos evaluating multimodal models on video QA, tracking, and temporal/audio localization across memory, abstraction, physics, and semantics skill areas.

What does a good Perception Test score look like?

Strong multimodal models score well above chance on multiple-choice QA (typically 50-65% accuracy), while tracking and localization scores vary more widely and remain a differentiator between models.

Who created Perception Test?

Perception Test was created by Viorica Pătrăucean and a large team of collaborators at Google DeepMind, published at NeurIPS 2023 (Datasets & Benchmarks track).

Perception Test — Phases

Perception Test