# GQA ![Category](https://img.shields.io/badge/Category-Reasoning-1f6feb) ![Metric](https://img.shields.io/badge/Metric-%25%20Accuracy-2da44e) ![Saturation](https://img.shields.io/badge/Saturation-High-d29922) ![Created](https://img.shields.io/badge/Created-Feb%202019-555) [![Paper](https://img.shields.io/badge/Paper-arXiv%201902.09506-b31b1b)](https://arxiv.org/abs/1902.09506) [![Dataset](https://img.shields.io/badge/Dataset-HuggingFace-ff9d00?logo=huggingface&logoColor=white)](https://huggingface.co/datasets/lmms-lab/GQA) > **Quick answer:** GQA (Hudson and Manning, 2019) is a visual question answering benchmark built from real-world scene graphs, designed to test compositional, multi-step spatial and relational reasoning rather than simple object recognition. North Micro Vision scores 57.4% on the `TestDev_Balanced` split as of August 2026. ## At a Glance **What it tests:** Compositional reasoning over real-world images — multi-step questions generated from scene graphs that require understanding objects, attributes, and relationships together. **Why it matters:** GQA's questions are programmatically generated from structured scene graphs with balanced answer distributions, reducing the language-prior shortcuts that let models "guess well" on earlier VQA datasets without truly reasoning over the image. **Known limitations:** As one of the older VQA benchmarks (2019), well-trained modern vision-language models increasingly saturate its `TestDev_Balanced` split, and its Visual Genome-derived scene graphs occasionally contain noisy annotations. ## What GQA Measures GQA pairs real-world images with questions synthesized from Visual Genome scene graphs — structured annotations of objects, attributes, and relationships. Each question is generated via a functional program over the scene graph, letting the benchmark systematically probe specific reasoning skills (spatial relations, comparisons, logical operations, and multi-hop composition) rather than relying on naturally-written, potentially biased questions. A key design goal was balancing answer distributions to reduce "language prior" exploits — earlier VQA datasets could be gamed by models that learned statistical shortcuts (e.g. answering "yes" more often for certain question phrasings) without genuinely grounding answers in the image. GQA's balanced `TestDev_Balanced` split is the most commonly reported evaluation configuration for this reason. Because GQA predates the current generation of frontier vision-language models by several years, it now functions more as a baseline sanity check and small-model discriminator than a frontier-model differentiator — strong models comfortably clear 70%+, while compact models like North Micro Vision land in the mid-50s to 60s. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Reasoning (compositional VQA) | | Metric | % accuracy | | Saturation | High | | Created by | Hudson and Manning | | Source paper | [Hudson and Manning 2019](https://arxiv.org/abs/1902.09506) | | Dataset | [lmms-lab/GQA on HuggingFace](https://huggingface.co/datasets/lmms-lab/GQA) | ## How GQA Is Scored Models answer open-ended or multiple-choice questions generated from scene-graph programs; the score is percent accuracy on the balanced `TestDev_Balanced` split, the standard configuration used to reduce language-prior gaming. ## State-of-the-Art Results | Rank | Model | Score | Source | Date | |---|---|---|---|---| | 1 | [North Micro Vision Instruct](/models/cohere-labs/north-micro-vision-instruct) | 57.4% | [North Micro Vision launch blog](https://huggingface.co/blog/CohereLabs/meet-north-micro-vision-instruct) | 2026-08 | *Score sourced from Cohere Labs' North Micro Vision Instruct launch announcement, August 2026, evaluated on the `TestDev_Balanced` split.* ## GQA on Benchgen > No Benchgen results yet — [be the first to run GQA](/benchmarks/hudson-and-manning/gqa). ## GQA vs Other Benchmarks | Benchmark | What it tests | Saturation | |---|---|---| | **GQA** | Compositional real-world scene reasoning | High | | [MMBench](/benchmarks/liu-et-al/mmbench) | ~20 fine-grained VQA ability dimensions, CircularEval | Medium | | [MMStar](/benchmarks/chen-et-al/mmstar) | Vision-indispensable, curated general VQA | Low | GQA remains useful as a well-understood, long-running compositional-reasoning check, but MMStar and MMBench are the more discriminative choices for comparing today's frontier and near-frontier vision-language models. ## Run GQA on Your Model Benchgen lets you evaluate your own vision-language model's compositional visual reasoning against GQA's balanced test split and track results across model versions. ## Related Models & Benchmarks - Top performers: [North Micro Vision Instruct](/models/cohere-labs/north-micro-vision-instruct) - Related benchmarks: [MMBench](/benchmarks/liu-et-al/mmbench) · [MMStar](/benchmarks/chen-et-al/mmstar) ## Frequently Asked Questions

What is GQA?

GQA is a visual question answering benchmark built from real-world scene graphs, testing compositional, multi-step spatial and relational reasoning over images.

What does a good GQA score look like?

North Micro Vision Instruct reports 57.4% on TestDev_Balanced as of August 2026; strong frontier vision-language models typically score in the 65-75% range.

Who created GQA?

GQA was created by Hudson and Manning; see the [original paper](https://arxiv.org/abs/1902.09506).

Is GQA saturated?

It's showing high saturation for larger models, though it remains a useful discriminator for compact/small vision-language models.

GQA — Results

GQA — Phases

GQA