POPE
1 phaseActive
Polling-based object hallucination benchmark for vision-language models using yes/no questions; created by Renmin University of China (EMNLP 2023).
Overview
POPE
Quick answer: POPE (Polling-based Object Probing Evaluation) is a benchmark that measures object hallucination in large vision-language models by polling them with simple yes/no questions about whether specific objects appear in an image, scored via F1.
At a Glance
What it tests: Whether a vision-language model reports objects that are not actually present in an image — the core "object hallucination" failure mode.
Why it matters: Prior hallucination evaluations were sensitive to prompt phrasing and generation style; POPE's polling-based yes/no format is more stable and lets researchers isolate hallucination from other capability gaps.
Known limitations: POPE only probes object existence, not attributes, relations, or counts, and frontier models now score highly on the random/popular sampling settings, leaving the adversarial setting as the main remaining signal.
What POPE Measures
POPE converts object hallucination evaluation into a simple polling task: for a given image, the model is asked a series of yes/no questions ("Is there a {object} in the image?") covering both objects that actually appear and objects that do not. Negative objects are sampled under three settings of increasing difficulty — random (objects sampled at random from the label set), popular (the most frequent objects in the dataset that don't appear in this image), and adversarial (objects that frequently co-occur with the image's true objects, making them easy to hallucinate).
The paper's central finding is that earlier generation-based hallucination metrics were unstable and sensitive to how questions were phrased, while POPE's controlled polling format produces consistent, reproducible rankings across models — motivating its use as a standard hallucination check alongside general VQA benchmarks.
POPE is built on top of existing captioned-image datasets (COCO, A-OKVQA, GQA), reusing their object annotations rather than requiring new image collection.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Multimodal (hallucination evaluation) |
| Metric | F1 score (%), also reporting accuracy, precision, recall, and yes-ratio |
| Number of tasks | ~9,000 yes/no polling questions across random, popular, and adversarial sampling settings |
| Saturation | High on random/popular settings; adversarial setting remains harder |
| Created by | Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, Ji-Rong Wen |
| Source paper | Li et al. 2023 (EMNLP 2023) |
| Code & data | GitHub |
How POPE Is Scored
For each image and sampling setting, the model answers a balanced set of yes/no questions (half with objects present, half absent). Responses are compared against ground truth to compute F1 score, the primary reported metric, along with accuracy and a "yes-ratio" diagnostic that flags models with a systematic bias toward answering "yes."
State-of-the-Art Results
Scores fluctuate by model release cadence and evaluation harness (prompt template, sampling setting). See the official repository for current standings across random, popular, and adversarial settings.
POPE on Benchgen
No Benchgen results yet — be the first to run POPE.
POPE vs Other Benchmarks
| Benchmark | What it tests | Tasks | Saturation |
|---|---|---|---|
| POPE | Object hallucination via yes/no polling | 9,000 | high |
| MME | Broader 14-subtask perception + cognition | — | high |
| MME-RealWorld | Harder, high-resolution real-world perception | — | low |
POPE's narrow, stable hallucination focus complements MME's broader 14-subtask evaluation, giving teams a clean, isolated signal on hallucination behavior alongside general perception scoring.
Run POPE on Your Model
Benchgen lets teams run POPE's random/popular/adversarial polling suites against their own model versions on demand, tracking F1 and yes-ratio over time to catch hallucination regressions before they reach production.
Related Models & Benchmarks
Explore related multimodal benchmarks like MME and MME-RealWorld to compare hallucination-focused and broader perception evaluation approaches.