RefCOCOg
1 phaseActive
Referring expression comprehension benchmark on MS-COCO images with longer natural-language descriptions, created by researchers at UCLA and Google.
Overview
RefCOCOg
Quick answer: RefCOCOg is a referring expression comprehension benchmark built on MS-COCO images, where models must localize the specific object described by a natural-language referring expression — RefCOCOg's expressions are notably longer and more descriptive than its sibling datasets RefCOCO and RefCOCO+.
At a Glance
What it tests: Visual grounding — given an image and a natural-language description of one object among several similar ones, can the model correctly identify (localize) that exact object?
Why it matters: RefCOCOg's longer, free-form expressions (averaging roughly twice the word count of RefCOCO/RefCOCO+) make it a harder, more naturalistic test of language grounding than shorter, appearance-only referring expressions.
Known limitations: Being built on MS-COCO's 80 object categories from 2015-era imagery, it is now a largely saturated benchmark for frontier grounding-capable VLMs.
What RefCOCOg Measures
RefCOCOg is one of three referring-expression datasets (alongside RefCOCO and RefCOCO+) built from MS-COCO images containing multiple instances of the same object category, forcing referring expressions to disambiguate between similar objects using spatial relations, attributes, or context. Unlike RefCOCO (collected in an interactive, time-pressured two-player game) and RefCOCO+ (which disallows location words like "left" or "right"), RefCOCOg was collected via a non-interactive Amazon Mechanical Turk setup that encourages longer, more natural, descriptive expressions.
Two standard splits are used in the literature: the "Google" split (from the original paper) and the "UMD" split, which fixes a testing leakage issue in the Google split and is now the more commonly reported evaluation split, containing 95,010 referring expressions for 54,822 objects across 26,711 images.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Multimodal (referring expression comprehension / visual grounding) |
| Metric | Grounding accuracy (%) — predicted bounding box counted correct if IoU ≥ 0.5 with ground truth |
| Number of tasks | 95,010 referring expressions for 54,822 objects across 26,711 images (UMD split) |
| Saturation | High — modern grounding-capable VLMs exceed 90% accuracy |
| Created by | Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan Yuille, Kevin Murphy |
| Source paper | Mao et al. 2016 (CVPR 2016) |
| Toolbox & data | GitHub |
How RefCOCOg Is Scored
For each referring expression, the model predicts a bounding box (or point/region) for the described object. A prediction is scored as correct if its Intersection-over-Union (IoU) with the ground-truth box is at least 0.5; overall accuracy is the percentage of correctly grounded expressions across the test set.
State-of-the-Art Results
Scores fluctuate by model release cadence and evaluation harness (prompt template, split used — Google vs UMD). See the dataset repository for reference splits and baseline comparisons.
RefCOCOg on Benchgen
No Benchgen results yet — be the first to run RefCOCOg.
RefCOCOg vs Other Benchmarks
| Benchmark | What it tests | Tasks | Saturation |
|---|---|---|---|
| RefCOCOg | Referring expression comprehension (long, descriptive) | 95,010 expressions | high |
| ScreenSpot | GUI element grounding from natural-language instructions | — | medium |
| PointGrounding | Point-level visual grounding | — | medium |
RefCOCOg's natural-image object grounding complements domain-specific grounding benchmarks like ScreenSpot, which tests the same underlying localization skill in GUI screenshots rather than everyday photos.
Run RefCOCOg on Your Model
Benchgen lets teams run RefCOCOg (Google and UMD splits) against their own model versions on demand, tracking grounding accuracy over time as a proxy for general visual localization ability.
Related Models & Benchmarks
Explore related grounding benchmarks like ScreenSpot to compare natural-image and GUI-based visual grounding evaluation approaches.