# VLMsAreBlind ![Category](https://img.shields.io/badge/Category-Multimodal-1f6feb) ![Metric](https://img.shields.io/badge/Metric-Accuracy-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-7-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-low-d29922) ![Created](https://img.shields.io/badge/Created-Jul%202024-555) [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/abs/2407.06581) [![GitHub](https://img.shields.io/badge/GitHub-Code-181717?logo=github&logoColor=white)](https://github.com/anguyen8/vision-llms-are-blind) > **Quick answer:** VLMsAreBlind (also called BlindTest) is a suite of 7 deliberately simple low-level vision tasks — like judging whether two circles touch or counting shapes in a logo — that expose a surprising gap between VLMs' benchmark scores and basic visual acuity that humans solve near-instantly. ## At a Glance **What it tests:** Precise low-level spatial perception — circle overlap, line intersection counting, circled-letter identification, and shape counting — using clean, unambiguous geometric images. **Why it matters:** VLMs scoring highly on broad benchmarks like MMMU can still fail tasks a human solves in seconds, revealing that strong aggregate scores can mask basic perceptual blind spots relevant to any production use case involving diagrams, UI screenshots, or spatial layout. **Known limitations:** The tasks are synthetic and narrow (geometric primitives only), so BlindTest measures a specific failure mode rather than general visual understanding — it complements broader multimodal benchmarks rather than replacing them. ## What VLMsAreBlind Measures VLMsAreBlind evaluates seven simple visual tasks: counting line intersections, determining whether two circles overlap, identifying which letter in a word is circled, counting circles in an Olympic-logo-style image, counting nested squares, counting rows and columns in a grid, and following color-coded paths on a subway-map-style diagram. Every task is designed so a human can answer correctly with near-100% accuracy at a glance. Despite this, the paper found four state-of-the-art VLMs (including GPT-4o and Gemini 1.5 Pro) averaged only 58% accuracy across tasks, with the best model (Claude 3.5 Sonnet) reaching just under 78% — far short of human-level performance. Linear-probing experiments showed the underlying vision encoders actually do capture the correct visual information; the failure occurs when the language model decodes that information into a verbal answer. The authors argue this reveals a structural weakness in how current VLM architectures translate detailed visual features into language, rather than a data or training-scale problem alone — a finding directly relevant to any production application relying on precise spatial reasoning. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Multimodal (low-level vision / spatial perception) | | Metric | Accuracy (%) across 7 tasks | | Number of tasks | 7 distinct low-level vision tasks | | Saturation | Low — even frontier VLMs fall well short of human accuracy | | Created by | Pooyan Rahmanzadehgervi, Logan Bolton, Mohammad Reza Taesiri, Anh Totti Nguyen | | Source paper | [Rahmanzadehgervi et al. 2024](https://arxiv.org/abs/2407.06581) | | Code | [GitHub](https://github.com/anguyen8/vision-llms-are-blind) | | Dataset | [Hugging Face](https://huggingface.co/datasets/XAI/vlmsareblind) | ## How VLMsAreBlind Is Scored Each of the 7 tasks is scored as a simple accuracy percentage (fraction of instances answered correctly), then averaged across tasks to produce an overall score. Because the correct answer is deterministic and unambiguous, there is no LLM-judge subjectivity involved. ## State-of-the-Art Results > Scores fluctuate with model version and image resolution/line-width settings. See the [official repository](https://github.com/anguyen8/vision-llms-are-blind) for current task-by-task standings. ## VLMsAreBlind on Benchgen > No Benchgen results yet — [be the first to run VLMsAreBlind](/benchmarks/rahmanzadehgervi-et-al-auburn-university/vlmsareblind). ## VLMsAreBlind vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **VLMsAreBlind** | Low-level spatial perception (counting, intersections, overlap) | 7 | low | | [MME](/benchmarks/chaoyou-fu-et-al-xiamen-university-tencent-youtu-lab/mme) | Broad perception + cognition, yes/no questions | 2,374 | high | | [POPE](/benchmarks/pope) | Object-existence hallucination | — | high | VLMsAreBlind is narrower and harder-edged than general perception suites like MME, isolating a specific class of spatial-reasoning failure that broader benchmarks tend to average away. ## Run VLMsAreBlind on Your Model Benchgen lets teams run the BlindTest suite against their own model versions on demand, tracking whether spatial-reasoning regressions creep in across releases — a failure mode that's invisible in aggregate multimodal scores but can break real UI-automation or diagram-reading use cases. ## Related Models & Benchmarks Explore related multimodal benchmarks like [MME](/benchmarks/chaoyou-fu-et-al-xiamen-university-tencent-youtu-lab/mme) and [POPE](/benchmarks/pope) to compare broad perception and hallucination evaluation with VLMsAreBlind's narrow spatial-acuity focus. ## Frequently Asked Questions

What is VLMsAreBlind?

VLMsAreBlind (BlindTest) is a 7-task benchmark of deliberately simple low-level vision problems — like counting circles or judging line intersections — that reveal a surprising gap between VLMs' benchmark scores and basic visual acuity.

What does a good VLMsAreBlind score look like?

Even the best-performing frontier model in the original paper (Claude 3.5 Sonnet) reached only ~78% accuracy, well below the near-100% humans achieve, so scores in the 70-80% range currently represent the state of the art.

Who created VLMsAreBlind?

VLMsAreBlind was created by Pooyan Rahmanzadehgervi, Logan Bolton, Mohammad Reza Taesiri, and Anh Totti Nguyen at Auburn University and the University of Alberta, presented as an Oral at ACCV 2024.

Rank	Model	Score
1	claude-3-5-sonnet	77.84
2	gemini-1-5-pro	58.48
3	gpt-4o	50.23
4	claude-3-sonnet	45.73

VLMsAreBlind — Results

VLMsAreBlind — Leaderboard

VLMsAreBlind Leaderboard