HallusionBench — Results
| Rank | Model | Score |
|---|---|---|
| 1 | north-micro-vision-instruct | 61.5 |
| 2 | lfm2-5-vl-3b | 47.2 |
HallusionBench
1 phaseActive
Diagnostic suite for entangled language hallucination and visual illusion in vision-language models — 346 images, 1,129 questions. Metric: question-pair accuracy.
Overview
HallusionBench
Quick answer: HallusionBench is a diagnostic benchmark that probes entangled language hallucination and visual illusion in vision-language models, using 346 images and 1,129 controlled questions, created by Guan et al. (2023). The best publicly verified question-pair accuracy is only 31.4% (GPT4V, human-graded), meaning even the strongest models fail most consistency checks — no modern frontier-model results have been publicly re-verified against the official leaderboard since.
At a Glance
What it tests: Whether a vision-language model actually grounds its answers in the image, or falls back on plausible-sounding language priors and misreads visual illusions. Why it matters: Many multimodal benchmarks only test single-question accuracy; HallusionBench's paired/controlled question design specifically catches models that answer confidently but inconsistently across near-identical visual conditions. Known limitations: Publicly available verified scores are limited to 2023-era models (GPT4V, Claude 3, LLaVA-1.5, and similar) — no modern frontier vision-language model scores could be independently verified for this page. Treat the table below as historical reference, not a current-generation leaderboard.
What HallusionBench Measures
HallusionBench is built around controlled image/question pairs designed to isolate two failure modes: language hallucination (the model answers based on prior knowledge or common patterns rather than the actual image) and visual illusion (the model misperceives the image itself, similar to how humans can be fooled by optical illusions). By pairing near-identical questions and images with subtly different ground truths, it can detect when a model is guessing or pattern-matching rather than genuinely reasoning over what it sees.
Benchmark Specifications
| Field | Value |
|---|---|
| Images | 346 |
| Questions | 1,129 |
| Task category | Multimodal / vision-language hallucination |
| Metric | Accuracy per figure, per question, and per question pair (question-pair is the headline metric) |
| Released | October 2023 |
| Created by | Tianrui Guan, Fuxiao Liu, and collaborators (University of Maryland, in collaboration with Tianyi Zhou's lab) |
How HallusionBench Is Scored
Models are graded on three levels: whether they get an individual question right, whether they get all questions right for a given figure, and — the headline metric — whether they answer an entire controlled question pair correctly, which is the strictest test of genuine visual grounding rather than lucky guesses. Grading can be done via exact-match or GPT-based evaluation of free-form answers.
State-of-the-Art Results
Caveat: These are the only publicly verified scores Benchgen could confirm directly from the official HallusionBench repository. They are all 2023-era models — no modern frontier vision-language model (e.g. current GPT, Claude, or Gemini generations) scores have been independently re-verified for this page. Readers should not treat this table as reflecting current-generation model capability.
| Rank | Model | Question-Pair Accuracy | Source |
|---|---|---|---|
| 1 | GPT4V (Sep 25, 2023 version, human eval) | 31.42% | HallusionBench repo |
| 2 | GPT4V (Sep 25, 2023 version, GPT eval) | 28.79% | HallusionBench repo |
| 3 | Claude 3 (GPT eval) | 21.76% | HallusionBench repo |
| 4 | GUA_VL (GPT eval) | 16.70% | HallusionBench repo |
| 5 | BLIP2-T5 (GPT eval) | 15.16% | HallusionBench repo |
| 6 | mPLUG_Owl-v2 (GPT eval) | 13.85% | HallusionBench repo |
| 7 | LLaVA-1.5 / MiniGPT5 (GPT eval) | 10.55% | HallusionBench repo |
| 9 | LLaVA-1.5 (human eval) / InstructBLIP (GPT eval) | 9.45% | HallusionBench repo |
HallusionBench on Benchgen
No Benchgen results yet — be the first to run HallusionBench, including on current-generation models.
HallusionBench vs Other Benchmarks
| Benchmark | What it tests | Saturation |
|---|---|---|
| HallusionBench | Language hallucination + visual illusion in VLMs | Low |
| MMMU | Multimodal college-level reasoning | — |
| ChartQA | Question answering over charts | — |
Run HallusionBench on Your Model
Because no current-generation frontier model has publicly verified HallusionBench scores, running it on Benchgen is a good way to get a first, version-controlled result for your model rather than relying on 2023-era comparisons.