# ERQA ![Category](https://img.shields.io/badge/Category-Reasoning-1f6feb) ![Metric](https://img.shields.io/badge/Metric-%25%20Accuracy-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-400-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-Low-d29922) ![Created](https://img.shields.io/badge/Created-Mar%202025-555) [![Paper](https://img.shields.io/badge/Paper-arXiv%202503.20020-b31b1b)](https://arxiv.org/abs/2503.20020) [![GitHub](https://img.shields.io/badge/GitHub-embodiedreasoning%2FERQA-181717?logo=github&logoColor=white)](https://github.com/embodiedreasoning/ERQA) > **Quick answer:** ERQA (Embodied Reasoning Question Answering, Google DeepMind, 2025) is a 400-question multiple-choice visual benchmark testing a model's ability to reason about the physical world — spatial relationships, object trajectories, multi-view consistency, and state estimation — released alongside the Gemini Robotics paper. Qwen3.8-Flash-Next scores 72.3% as of August 2026. ## At a Glance **What it tests:** Whether a multimodal model can answer visual multiple-choice questions requiring embodied/physical-world reasoning — spatial relationships, action/trajectory reasoning, state estimation, and multi-view reasoning across images — rather than purely linguistic or static-scene understanding. **Why it matters:** Most multimodal QA benchmarks test static-scene perception or language-grounded reasoning; ERQA specifically probes reasoning skills relevant to embodied agents and robots operating in the physical world, released as part of Google DeepMind's Gemini Robotics work to measure whether a model's visual reasoning actually transfers to real-world physical interaction. **Known limitations:** At only 400 questions, ERQA is a relatively small evaluation set, so individual-question variance can matter more than on larger benchmarks; it is also newer (2025) and has fewer third-party-tracked model scores than longer-established multimodal benchmarks. ## What ERQA Measures ERQA evaluates multimodal models on 400 multiple-choice visual questions spanning categories such as spatial reasoning, trajectory reasoning, action reasoning, state estimation, and multi-view reasoning — the kinds of visual-world understanding an embodied agent (e.g. a robot) needs before it can act physically on what it perceives. It was released by the Gemini Robotics Team at Google DeepMind alongside their "Gemini Robotics: Bringing AI into the Physical World" paper, as a way to isolate and measure the embodied-reasoning capability of a vision-language model independent of the robot-control stack built on top of it. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Reasoning (embodied/physical-world) | | Metric | % accuracy | | Number of tasks | 400 multiple-choice questions | | Saturation | Low | | Created by | Gemini Robotics Team (Google DeepMind) | | Source paper | [Gemini Robotics Team et al. 2025](https://arxiv.org/abs/2503.20020) | | GitHub | [embodiedreasoning/ERQA](https://github.com/embodiedreasoning/ERQA) | ## How ERQA Is Scored Each of the 400 multiple-choice questions is scored correct or incorrect against a fixed ground-truth answer; the overall score is the percentage of questions answered correctly. ## State-of-the-Art Results | Rank | Model | Score | Source | Date | |---|---|---|---|---| | 1 | [Qwen3.8-Flash-Next](/models/qwen/qwen3-8-flash-next) | 72.3% | Qwen3.8-Flash-Next model card | 2026-08 | | 2 | [Qwen3.8-27B](/models/qwen/qwen3-8-27b) | 65.5% | Qwen3.8-27B model card | 2026-08 | *Scores above are self-reported by Alibaba's Qwen Team on the official Hugging Face model cards and shown for context. They are not Benchgen measurements, and are not drawn from a shared third-party leaderboard — only the 2 scores above are currently tracked locally. Both scores are independently cross-confirmed against llm-stats.com/benchmarks/erqa, which tracks 28 models on this benchmark.* ## ERQA on Benchgen > No Benchgen results yet — [be the first to run ERQA](/benchmarks/google-deepmind/erqa). ## ERQA vs Other Benchmarks | Benchmark | What it tests | Saturation | |---|---|---| | **ERQA** | Embodied/physical-world visual reasoning (spatial, trajectory, multi-view) | Low | | [AndroidWorld](/benchmarks/google-deepmind/androidworld) | Dynamic, parameterized mobile-use agent control | Low | | [OSWorld-Verified](/benchmarks/osworld/osworld-verified) | Real desktop/web computer-use agent control | — | ERQA is a static visual-QA probe of embodied reasoning, distinct from the live-environment agent benchmarks like AndroidWorld or OSWorld-Verified — use it to assess whether a model's visual understanding of physical scenes (spatial relationships, trajectories, multi-view consistency) is sound, independent of whether it can also execute multi-step actions in a live environment. ## Run ERQA on Your Model Benchgen lets you evaluate your own multimodal model against ERQA's 400 embodied-reasoning questions and track results across model versions. ## Related Models & Benchmarks - Top performers: [Qwen3.8-Flash-Next](/models/qwen/qwen3-8-flash-next) · [Qwen3.8-27B](/models/qwen/qwen3-8-27b) - Related benchmarks: [AndroidWorld](/benchmarks/google-deepmind/androidworld) · [OSWorld-Verified](/benchmarks/osworld/osworld-verified) ## Frequently Asked Questions

What is ERQA?

ERQA (Embodied Reasoning Question Answering) is a Google DeepMind benchmark released alongside the Gemini Robotics paper (2025) that tests multimodal models on 400 multiple-choice visual questions requiring physical-world reasoning — spatial relationships, trajectories, state estimation, and multi-view consistency.

What does a good ERQA score look like?

Qwen3.8-Flash-Next reports 72.3% as of August 2026, the highest currently tracked on Benchgen; Qwen3.8-27B scores 65.5%.

Who created ERQA?

ERQA was created by the Gemini Robotics Team at Google DeepMind; see the [original paper](https://arxiv.org/abs/2503.20020).

Is ERQA saturated?

No — ERQA remains a low-saturation benchmark; its 400 embodied-reasoning questions probing spatial/trajectory/multi-view understanding keep meaningful headroom between models.

How does ERQA differ from AndroidWorld?

ERQA is a static visual multiple-choice QA benchmark testing embodied reasoning about physical scenes, while AndroidWorld is a live, dynamic agent-control environment testing multi-step task execution on a real Android device — ERQA probes understanding, AndroidWorld probes action.

--- *Benchmark definition paraphrased from [Gemini Robotics Team et al. 2025](https://arxiv.org/abs/2503.20020). State-of-the-art scores sourced from published model cards and attributed inline. Last updated 2026-10-31.*

Rank	Model	Score
1	qwen3-8-flash-next	72.3
2	qwen3-8-27b	65.5

ERQA — Results

ERQA — Phases

ERQA