# RefCOCOg ![Category](https://img.shields.io/badge/Category-Multimodal-1f6feb) ![Metric](https://img.shields.io/badge/Metric-Grounding%20Accuracy-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-95%2C010%20expressions-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-high-d29922) ![Created](https://img.shields.io/badge/Created-Nov%202015-555) [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/abs/1511.02283) [![GitHub](https://img.shields.io/badge/GitHub-Toolbox-181717?logo=github&logoColor=white)](https://github.com/lichengunc/refer) > **Quick answer:** RefCOCOg is a referring expression comprehension benchmark built on MS-COCO images, where models must localize the specific object described by a natural-language referring expression — RefCOCOg's expressions are notably longer and more descriptive than its sibling datasets RefCOCO and RefCOCO+. ## At a Glance **What it tests:** Visual grounding — given an image and a natural-language description of one object among several similar ones, can the model correctly identify (localize) that exact object? **Why it matters:** RefCOCOg's longer, free-form expressions (averaging roughly twice the word count of RefCOCO/RefCOCO+) make it a harder, more naturalistic test of language grounding than shorter, appearance-only referring expressions. **Known limitations:** Being built on MS-COCO's 80 object categories from 2015-era imagery, it is now a largely saturated benchmark for frontier grounding-capable VLMs. ## What RefCOCOg Measures RefCOCOg is one of three referring-expression datasets (alongside RefCOCO and RefCOCO+) built from MS-COCO images containing multiple instances of the same object category, forcing referring expressions to disambiguate between similar objects using spatial relations, attributes, or context. Unlike RefCOCO (collected in an interactive, time-pressured two-player game) and RefCOCO+ (which disallows location words like "left" or "right"), RefCOCOg was collected via a non-interactive Amazon Mechanical Turk setup that encourages longer, more natural, descriptive expressions. Two standard splits are used in the literature: the "Google" split (from the original paper) and the "UMD" split, which fixes a testing leakage issue in the Google split and is now the more commonly reported evaluation split, containing 95,010 referring expressions for 54,822 objects across 26,711 images. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Multimodal (referring expression comprehension / visual grounding) | | Metric | Grounding accuracy (%) — predicted bounding box counted correct if IoU ≥ 0.5 with ground truth | | Number of tasks | 95,010 referring expressions for 54,822 objects across 26,711 images (UMD split) | | Saturation | High — modern grounding-capable VLMs exceed 90% accuracy | | Created by | Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan Yuille, Kevin Murphy | | Source paper | [Mao et al. 2016](https://arxiv.org/abs/1511.02283) (CVPR 2016) | | Toolbox & data | [GitHub](https://github.com/lichengunc/refer) | ## How RefCOCOg Is Scored For each referring expression, the model predicts a bounding box (or point/region) for the described object. A prediction is scored as correct if its Intersection-over-Union (IoU) with the ground-truth box is at least 0.5; overall accuracy is the percentage of correctly grounded expressions across the test set. ## State-of-the-Art Results > Scores fluctuate by model release cadence and evaluation harness (prompt template, split used — Google vs UMD). See the [dataset repository](https://github.com/lichengunc/refer) for reference splits and baseline comparisons. ## RefCOCOg on Benchgen > No Benchgen results yet — [be the first to run RefCOCOg](/benchmarks/junhua-mao-jonathan-huang-alexander-toshev-oana-camburu-alan-yuille-kevin-murphy-ucla-google/refcocog). ## RefCOCOg vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **RefCOCOg** | Referring expression comprehension (long, descriptive) | 95,010 expressions | high | | [ScreenSpot](/benchmarks/screenspot) | GUI element grounding from natural-language instructions | — | medium | | [PointGrounding](/benchmarks/pointgrounding) | Point-level visual grounding | — | medium | RefCOCOg's natural-image object grounding complements domain-specific grounding benchmarks like ScreenSpot, which tests the same underlying localization skill in GUI screenshots rather than everyday photos. ## Run RefCOCOg on Your Model Benchgen lets teams run RefCOCOg (Google and UMD splits) against their own model versions on demand, tracking grounding accuracy over time as a proxy for general visual localization ability. ## Related Models & Benchmarks Explore related grounding benchmarks like [ScreenSpot](/benchmarks/screenspot) to compare natural-image and GUI-based visual grounding evaluation approaches. ## Frequently Asked Questions

What is RefCOCOg?

RefCOCOg is a referring expression comprehension benchmark built on MS-COCO images, requiring models to localize the object described by a longer, more descriptive natural-language expression than its sibling datasets RefCOCO and RefCOCO+.

What does a good RefCOCOg score look like?

Modern grounding-capable vision-language models exceed 90% accuracy (IoU@0.5) on RefCOCOg, reflecting significant saturation on this decade-old benchmark.

Who created RefCOCOg?

RefCOCOg was created by Junhua Mao and collaborators at UCLA and Google, published at CVPR 2016.

RefCOCOg — Phases

RefCOCOg