ERQA

ERQA

Quick answer: ERQA (Embodied Reasoning Question Answering, Google DeepMind, 2025) is a 400-question multiple-choice visual benchmark testing a model's ability to reason about the physical world — spatial relationships, object trajectories, multi-view consistency, and state estimation — released alongside the Gemini Robotics paper. Qwen3.8-Flash-Next scores 72.3% as of August 2026.

At a Glance

What it tests: Whether a multimodal model can answer visual multiple-choice questions requiring embodied/physical-world reasoning — spatial relationships, action/trajectory reasoning, state estimation, and multi-view reasoning across images — rather than purely linguistic or static-scene understanding.

Why it matters: Most multimodal QA benchmarks test static-scene perception or language-grounded reasoning; ERQA specifically probes reasoning skills relevant to embodied agents and robots operating in the physical world, released as part of Google DeepMind's Gemini Robotics work to measure whether a model's visual reasoning actually transfers to real-world physical interaction.

Known limitations: At only 400 questions, ERQA is a relatively small evaluation set, so individual-question variance can matter more than on larger benchmarks; it is also newer (2025) and has fewer third-party-tracked model scores than longer-established multimodal benchmarks.

What ERQA Measures

ERQA evaluates multimodal models on 400 multiple-choice visual questions spanning categories such as spatial reasoning, trajectory reasoning, action reasoning, state estimation, and multi-view reasoning — the kinds of visual-world understanding an embodied agent (e.g. a robot) needs before it can act physically on what it perceives. It was released by the Gemini Robotics Team at Google DeepMind alongside their "Gemini Robotics: Bringing AI into the Physical World" paper, as a way to isolate and measure the embodied-reasoning capability of a vision-language model independent of the robot-control stack built on top of it.

Benchmark Specifications

How ERQA Is Scored

Each of the 400 multiple-choice questions is scored correct or incorrect against a fixed ground-truth answer; the overall score is the percentage of questions answered correctly.

State-of-the-Art Results

Scores above are self-reported by Alibaba's Qwen Team on the official Hugging Face model cards and shown for context. They are not Benchgen measurements, and are not drawn from a shared third-party leaderboard — only the 2 scores above are currently tracked locally. Both scores are independently cross-confirmed against llm-stats.com/benchmarks/erqa, which tracks 28 models on this benchmark.

ERQA on Benchgen

No Benchgen results yet — be the first to run ERQA.

ERQA vs Other Benchmarks

ERQA is a static visual-QA probe of embodied reasoning, distinct from the live-environment agent benchmarks like AndroidWorld or OSWorld-Verified — use it to assess whether a model's visual understanding of physical scenes (spatial relationships, trajectories, multi-view consistency) is sound, independent of whether it can also execute multi-step actions in a live environment.

Run ERQA on Your Model

Benchgen lets you evaluate your own multimodal model against ERQA's 400 embodied-reasoning questions and track results across model versions.

Related Models & Benchmarks

• Top performers: Qwen3.8-Flash-Next · Qwen3.8-27B
• Related benchmarks: AndroidWorld · OSWorld-Verified

Frequently Asked Questions

Benchmark definition paraphrased from Gemini Robotics Team et al. 2025. State-of-the-art scores sourced from published model cards and attributed inline. Last updated 2026-10-31.