# POPE ![Category](https://img.shields.io/badge/Category-Multimodal-1f6feb) ![Metric](https://img.shields.io/badge/Metric-F1%20Score-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-9%2C000-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-high-d29922) ![Created](https://img.shields.io/badge/Created-May%202023-555) [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/abs/2305.10355) [![GitHub](https://img.shields.io/badge/GitHub-Repo-181717?logo=github&logoColor=white)](https://github.com/RUCAIBox/POPE) > **Quick answer:** POPE (Polling-based Object Probing Evaluation) is a benchmark that measures object hallucination in large vision-language models by polling them with simple yes/no questions about whether specific objects appear in an image, scored via F1. ## At a Glance **What it tests:** Whether a vision-language model reports objects that are not actually present in an image — the core "object hallucination" failure mode. **Why it matters:** Prior hallucination evaluations were sensitive to prompt phrasing and generation style; POPE's polling-based yes/no format is more stable and lets researchers isolate hallucination from other capability gaps. **Known limitations:** POPE only probes object *existence*, not attributes, relations, or counts, and frontier models now score highly on the random/popular sampling settings, leaving the adversarial setting as the main remaining signal. ## What POPE Measures POPE converts object hallucination evaluation into a simple polling task: for a given image, the model is asked a series of yes/no questions ("Is there a {object} in the image?") covering both objects that actually appear and objects that do not. Negative objects are sampled under three settings of increasing difficulty — **random** (objects sampled at random from the label set), **popular** (the most frequent objects in the dataset that don't appear in this image), and **adversarial** (objects that frequently co-occur with the image's true objects, making them easy to hallucinate). The paper's central finding is that earlier generation-based hallucination metrics were unstable and sensitive to how questions were phrased, while POPE's controlled polling format produces consistent, reproducible rankings across models — motivating its use as a standard hallucination check alongside general VQA benchmarks. POPE is built on top of existing captioned-image datasets (COCO, A-OKVQA, GQA), reusing their object annotations rather than requiring new image collection. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Multimodal (hallucination evaluation) | | Metric | F1 score (%), also reporting accuracy, precision, recall, and yes-ratio | | Number of tasks | ~9,000 yes/no polling questions across random, popular, and adversarial sampling settings | | Saturation | High on random/popular settings; adversarial setting remains harder | | Created by | Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, Ji-Rong Wen | | Source paper | [Li et al. 2023](https://arxiv.org/abs/2305.10355) (EMNLP 2023) | | Code & data | [GitHub](https://github.com/RUCAIBox/POPE) | ## How POPE Is Scored For each image and sampling setting, the model answers a balanced set of yes/no questions (half with objects present, half absent). Responses are compared against ground truth to compute F1 score, the primary reported metric, along with accuracy and a "yes-ratio" diagnostic that flags models with a systematic bias toward answering "yes." ## State-of-the-Art Results > Scores fluctuate by model release cadence and evaluation harness (prompt template, sampling setting). See the [official repository](https://github.com/RUCAIBox/POPE) for current standings across random, popular, and adversarial settings. ## POPE on Benchgen > No Benchgen results yet — [be the first to run POPE](/benchmarks/yifan-li-yifan-du-kun-zhou-jinpeng-wang-wayne-xin-zhao-ji-rong-wen-renmin-university-of-china/pope). ## POPE vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **POPE** | Object hallucination via yes/no polling | 9,000 | high | | [MME](/benchmarks/mme) | Broader 14-subtask perception + cognition | — | high | | [MME-RealWorld](/benchmarks/mme-realworld) | Harder, high-resolution real-world perception | — | low | POPE's narrow, stable hallucination focus complements MME's broader 14-subtask evaluation, giving teams a clean, isolated signal on hallucination behavior alongside general perception scoring. ## Run POPE on Your Model Benchgen lets teams run POPE's random/popular/adversarial polling suites against their own model versions on demand, tracking F1 and yes-ratio over time to catch hallucination regressions before they reach production. ## Related Models & Benchmarks Explore related multimodal benchmarks like [MME](/benchmarks/mme) and [MME-RealWorld](/benchmarks/mme-realworld) to compare hallucination-focused and broader perception evaluation approaches. ## Frequently Asked Questions

What is POPE?

POPE (Polling-based Object Probing Evaluation) is a benchmark that measures object hallucination in vision-language models using ~9,000 yes/no polling questions across random, popular, and adversarial sampling settings.

What does a good POPE score look like?

Strong models score in the high 80s to low 90s F1 on the random and popular settings; the adversarial setting is harder and better separates models with weaker hallucination robustness.

Who created POPE?

POPE was created by Yifan Li and collaborators at Renmin University of China, published at EMNLP 2023.

POPE — Phases

POPE