# ERQA ![Category](https://img.shields.io/badge/Category-Reasoning-1f6feb) ![Metric](https://img.shields.io/badge/Metric-%25%20Accuracy-2da44e) ![Questions](https://img.shields.io/badge/Questions-400-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-Low-d29922) ![Created](https://img.shields.io/badge/Created-Mar%202025-555) [![Paper](https://img.shields.io/badge/Paper-arXiv%202503.20020-b31b1b)](https://arxiv.org/abs/2503.20020) [![GitHub](https://img.shields.io/badge/GitHub-embodiedreasoning%2FERQA-181717?logo=github&logoColor=white)](https://github.com/embodiedreasoning/ERQA) > **Quick answer:** ERQA (Embodied Reasoning Question Answering, Gemini Robotics Team, Google DeepMind, 2025) is a 400-question multiple-choice visual QA benchmark covering spatial reasoning, trajectory reasoning, action reasoning, state estimation, and multi-view reasoning — the embodied-reasoning skills needed for physical-world agents. Qwen3.8-Flash-Next scores 72.3% as of August 2026. ## At a Glance **What it tests:** Whether a multimodal model can answer embodied-reasoning visual questions — understanding spatial relationships, predicting/interpreting object and agent trajectories, reasoning about physical actions, estimating state, and integrating information across multiple camera views. **Why it matters:** Introduced alongside Google DeepMind's Gemini Robotics release, ERQA isolates the embodied-reasoning capability that underlies real-world robotics and physical-agent tasks, independent of actual robot control — a clean multiple-choice diagnostic for how well a model understands physical scenes before it ever has to act on them. **Known limitations:** It is a multiple-choice QA format, so it measures embodied-reasoning *understanding*, not closed-loop robot control performance; with only 400 questions, individual question-type breakdowns carry wider variance than larger benchmarks. ## What ERQA Measures ERQA evaluates multimodal models on 400 multiple-choice visual questions spanning five embodied-reasoning categories: spatial reasoning, trajectory reasoning, action reasoning, state estimation, and multi-view reasoning. It was introduced as part of Google DeepMind's "Gemini Robotics: Bringing AI into the Physical World" release to measure the embodied-reasoning foundation needed for robotics and physical-world agents, decoupled from actual robot actuation. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Reasoning (embodied/multimodal) | | Metric | % accuracy | | Number of questions | 400 multiple-choice | | Saturation | Low | | Created by | Gemini Robotics Team (Google DeepMind) | | Source paper | [Gemini Robotics Team et al. 2025](https://arxiv.org/abs/2503.20020) | | GitHub | [embodiedreasoning/ERQA](https://github.com/embodiedreasoning/ERQA) | ## How ERQA Is Scored Models are scored on the percentage of the 400 multiple-choice embodied-reasoning questions answered correctly, covering spatial reasoning, trajectory reasoning, action reasoning, state estimation, and multi-view reasoning. ## State-of-the-Art Results | Rank | Model | Score | Source | Date | |---|---|---|---|---| | 1 | [Qwen3.8-Flash-Next](/models/qwen/qwen3-8-flash-next) | 72.3% | Qwen3.8-Flash-Next model card | 2026-08 | | 2 | [Qwen3.8-27B](/models/qwen/qwen3-8-27b) | 65.5% | Qwen3.8-27B model card | 2026-08 | *Scores above are self-reported by Alibaba's Qwen Team on the official Hugging Face model cards and shown for context. They are not Benchgen measurements, and are not drawn from a shared third-party leaderboard — only the 2 scores above are currently tracked locally.* ## ERQA on Benchgen > No Benchgen results yet — [be the first to run ERQA](/benchmarks/google-deepmind/erqa). ## ERQA vs Other Benchmarks | Benchmark | What it tests | Saturation | |---|---|---| | **ERQA** | Embodied reasoning (spatial/trajectory/action/state/multi-view QA) | Low | | [MMBench](/benchmarks/liu-et-al/mmbench) | General vision-language reasoning across 20 ability dimensions | Medium | | [AndroidWorld](/benchmarks/google-deepmind/androidworld) | Mobile-use agent control (live device state) | Low | ERQA is narrower and more diagnostic than general VQA benchmarks like MMBench — it specifically isolates embodied/physical-world reasoning, making it a useful companion check alongside agent-control benchmarks like AndroidWorld for models aimed at robotics or physical-agent use cases. ## Run ERQA on Your Model Benchgen lets you evaluate your own model against ERQA's 400 embodied-reasoning multiple-choice questions and track results across model versions. ## Related Models & Benchmarks - Top performers: [Qwen3.8-Flash-Next](/models/qwen/qwen3-8-flash-next) · [Qwen3.8-27B](/models/qwen/qwen3-8-27b) - Related benchmarks: [MMBench](/benchmarks/liu-et-al/mmbench) · [AndroidWorld](/benchmarks/google-deepmind/androidworld) ## Frequently Asked Questions

What is ERQA?

ERQA (Embodied Reasoning Question Answering) is a Google DeepMind benchmark introduced with the Gemini Robotics release (2025), testing embodied reasoning via 400 multiple-choice visual questions across spatial, trajectory, action, state-estimation, and multi-view reasoning.

What does a good ERQA score look like?

Qwen3.8-Flash-Next reports 72.3% as of August 2026, the highest currently tracked on Benchgen; Qwen3.8-27B scores 65.5%.

Who created ERQA?

ERQA was created by the Gemini Robotics Team at Google DeepMind; see the [original paper](https://arxiv.org/abs/2503.20020).

Is ERQA saturated?

No — ERQA remains a low-saturation benchmark; current top scores sit in the low-to-mid 70s percent, leaving substantial headroom for embodied-reasoning improvement.

How does ERQA differ from AndroidWorld?

ERQA is a static multiple-choice QA benchmark testing embodied-reasoning *understanding* (spatial/trajectory/action/state/multi-view), while AndroidWorld tests live, closed-loop agent *control* of a real Android device — ERQA measures the reasoning foundation, AndroidWorld measures acting on it.

--- *Benchmark definition paraphrased from [Gemini Robotics Team et al. 2025](https://arxiv.org/abs/2503.20020). State-of-the-art scores sourced from published model cards and attributed inline. Last updated 2026-10-31.*

Rank	Model	Score
1	qwen3-8-flash-next	72.3
2	qwen3-8-27b	65.5

ERQA — Results

ERQA — Evaluations