# HallusionBench ![Category](https://img.shields.io/badge/Category-Multimodal-1f6feb) ![Metric](https://img.shields.io/badge/Metric-Question--Pair%20Accuracy-2da44e) ![Questions](https://img.shields.io/badge/Questions-1%2C129-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-Low-red) ![Created](https://img.shields.io/badge/Created-Oct%202023-555) [![Paper](https://img.shields.io/badge/Paper-arXiv%202310.14566-b31b1b)](https://arxiv.org/abs/2310.14566) [![GitHub](https://img.shields.io/badge/GitHub-tianyi--lab%2FHallusionBench-181717?logo=github&logoColor=white)](https://github.com/tianyi-lab/HallusionBench) > **Quick answer:** HallusionBench is a diagnostic benchmark that probes entangled language hallucination and visual illusion in vision-language models, using 346 images and 1,129 controlled questions, created by Guan et al. (2023). The best publicly verified question-pair accuracy is only 31.4% (GPT4V, human-graded), meaning even the strongest models fail most consistency checks — no modern frontier-model results have been publicly re-verified against the official leaderboard since. --- ## At a Glance **What it tests:** Whether a vision-language model actually grounds its answers in the image, or falls back on plausible-sounding language priors and misreads visual illusions. **Why it matters:** Many multimodal benchmarks only test single-question accuracy; HallusionBench's paired/controlled question design specifically catches models that answer confidently but inconsistently across near-identical visual conditions. **Known limitations:** Publicly available verified scores are limited to 2023-era models (GPT4V, Claude 3, LLaVA-1.5, and similar) — no modern frontier vision-language model scores could be independently verified for this page. Treat the table below as historical reference, not a current-generation leaderboard. ## What HallusionBench Measures HallusionBench is built around controlled image/question pairs designed to isolate two failure modes: **language hallucination** (the model answers based on prior knowledge or common patterns rather than the actual image) and **visual illusion** (the model misperceives the image itself, similar to how humans can be fooled by optical illusions). By pairing near-identical questions and images with subtly different ground truths, it can detect when a model is guessing or pattern-matching rather than genuinely reasoning over what it sees. ## Benchmark Specifications | Field | Value | |---|---| | Images | 346 | | Questions | 1,129 | | Task category | Multimodal / vision-language hallucination | | Metric | Accuracy per figure, per question, and per question pair (question-pair is the headline metric) | | Released | October 2023 | | Created by | Tianrui Guan, Fuxiao Liu, and collaborators (University of Maryland, in collaboration with Tianyi Zhou's lab) | ## How HallusionBench Is Scored Models are graded on three levels: whether they get an individual question right, whether they get all questions right for a given figure, and — the headline metric — whether they answer an entire controlled question *pair* correctly, which is the strictest test of genuine visual grounding rather than lucky guesses. Grading can be done via exact-match or GPT-based evaluation of free-form answers. ## State-of-the-Art Results > **Caveat:** These are the only publicly verified scores Benchgen could confirm directly from the [official HallusionBench repository](https://github.com/tianyi-lab/HallusionBench). They are all 2023-era models — no modern frontier vision-language model (e.g. current GPT, Claude, or Gemini generations) scores have been independently re-verified for this page. Readers should not treat this table as reflecting current-generation model capability. | Rank | Model | Question-Pair Accuracy | Source | |---|---|---:|---| | 1 | GPT4V (Sep 25, 2023 version, human eval) | 31.42% | [HallusionBench repo](https://github.com/tianyi-lab/HallusionBench) | | 2 | GPT4V (Sep 25, 2023 version, GPT eval) | 28.79% | [HallusionBench repo](https://github.com/tianyi-lab/HallusionBench) | | 3 | Claude 3 (GPT eval) | 21.76% | [HallusionBench repo](https://github.com/tianyi-lab/HallusionBench) | | 4 | GUA_VL (GPT eval) | 16.70% | [HallusionBench repo](https://github.com/tianyi-lab/HallusionBench) | | 5 | BLIP2-T5 (GPT eval) | 15.16% | [HallusionBench repo](https://github.com/tianyi-lab/HallusionBench) | | 6 | mPLUG_Owl-v2 (GPT eval) | 13.85% | [HallusionBench repo](https://github.com/tianyi-lab/HallusionBench) | | 7 | LLaVA-1.5 / MiniGPT5 (GPT eval) | 10.55% | [HallusionBench repo](https://github.com/tianyi-lab/HallusionBench) | | 9 | LLaVA-1.5 (human eval) / InstructBLIP (GPT eval) | 9.45% | [HallusionBench repo](https://github.com/tianyi-lab/HallusionBench) | ## HallusionBench on Benchgen > No Benchgen results yet — [be the first to run HallusionBench](/benchmarks/university-of-maryland/hallusionbench), including on current-generation models. ## HallusionBench vs Other Benchmarks | Benchmark | What it tests | Saturation | |---|---|---| | HallusionBench | Language hallucination + visual illusion in VLMs | Low | | [MMMU](/benchmarks/yue-et-al/mmmu) | Multimodal college-level reasoning | — | | [ChartQA](/benchmarks/vis-nlp/chartqa) | Question answering over charts | — | ## Run HallusionBench on Your Model Because no current-generation frontier model has publicly verified HallusionBench scores, running it on Benchgen is a good way to get a first, version-controlled result for your model rather than relying on 2023-era comparisons. ## Related Models & Benchmarks - [MMMU](/benchmarks/yue-et-al/mmmu) - [ChartQA](/benchmarks/vis-nlp/chartqa) ## Frequently Asked Questions

What is HallusionBench?

HallusionBench is a diagnostic benchmark for vision-language models that isolates two failure modes — language hallucination and visual illusion — using controlled image/question pairs across 346 images and 1,129 questions.

What does a good score look like?

The best verified question-pair accuracy on record is only 31.4% (GPT4V, human-graded), so the benchmark remains far from saturated — there's substantial headroom for modern models.

Who created HallusionBench?

HallusionBench was created by Tianrui Guan, Fuxiao Liu, and collaborators, published in October 2023.

Rank	Model	Score
1	lfm3-vl-2b	65.6
2	north-micro-vision-instruct	61.5
3	lfm2-5-vl-3b	47.2

HallusionBench — Results

HallusionBench — Evaluations