GQA

GQA

Quick answer: GQA (Hudson and Manning, 2019) is a visual question answering benchmark built from real-world scene graphs, designed to test compositional, multi-step spatial and relational reasoning rather than simple object recognition. North Micro Vision scores 57.4% on the TestDev_Balanced split as of August 2026.

At a Glance

What it tests: Compositional reasoning over real-world images — multi-step questions generated from scene graphs that require understanding objects, attributes, and relationships together.

Why it matters: GQA's questions are programmatically generated from structured scene graphs with balanced answer distributions, reducing the language-prior shortcuts that let models "guess well" on earlier VQA datasets without truly reasoning over the image.

Known limitations: As one of the older VQA benchmarks (2019), well-trained modern vision-language models increasingly saturate its TestDev_Balanced split, and its Visual Genome-derived scene graphs occasionally contain noisy annotations.

What GQA Measures

GQA pairs real-world images with questions synthesized from Visual Genome scene graphs — structured annotations of objects, attributes, and relationships. Each question is generated via a functional program over the scene graph, letting the benchmark systematically probe specific reasoning skills (spatial relations, comparisons, logical operations, and multi-hop composition) rather than relying on naturally-written, potentially biased questions.

A key design goal was balancing answer distributions to reduce "language prior" exploits — earlier VQA datasets could be gamed by models that learned statistical shortcuts (e.g. answering "yes" more often for certain question phrasings) without genuinely grounding answers in the image. GQA's balanced TestDev_Balanced split is the most commonly reported evaluation configuration for this reason.

Because GQA predates the current generation of frontier vision-language models by several years, it now functions more as a baseline sanity check and small-model discriminator than a frontier-model differentiator — strong models comfortably clear 70%+, while compact models like North Micro Vision land in the mid-50s to 60s.

Benchmark Specifications

How GQA Is Scored

Models answer open-ended or multiple-choice questions generated from scene-graph programs; the score is percent accuracy on the balanced TestDev_Balanced split, the standard configuration used to reduce language-prior gaming.

State-of-the-Art Results

Score sourced from Cohere Labs' North Micro Vision Instruct launch announcement, August 2026, evaluated on the TestDev_Balanced split.

GQA on Benchgen

No Benchgen results yet — be the first to run GQA.

GQA vs Other Benchmarks

GQA remains useful as a well-understood, long-running compositional-reasoning check, but MMStar and MMBench are the more discriminative choices for comparing today's frontier and near-frontier vision-language models.

Run GQA on Your Model

Benchgen lets you evaluate your own vision-language model's compositional visual reasoning against GQA's balanced test split and track results across model versions.

Related Models & Benchmarks

• Top performers: North Micro Vision Instruct
• Related benchmarks: MMBench · MMStar

Frequently Asked Questions