VLMsAreBlind

VLMsAreBlind

Quick answer: VLMsAreBlind (also called BlindTest) is a suite of 7 deliberately simple low-level vision tasks — like judging whether two circles touch or counting shapes in a logo — that expose a surprising gap between VLMs' benchmark scores and basic visual acuity that humans solve near-instantly.

At a Glance

What it tests: Precise low-level spatial perception — circle overlap, line intersection counting, circled-letter identification, and shape counting — using clean, unambiguous geometric images.

Why it matters: VLMs scoring highly on broad benchmarks like MMMU can still fail tasks a human solves in seconds, revealing that strong aggregate scores can mask basic perceptual blind spots relevant to any production use case involving diagrams, UI screenshots, or spatial layout.

Known limitations: The tasks are synthetic and narrow (geometric primitives only), so BlindTest measures a specific failure mode rather than general visual understanding — it complements broader multimodal benchmarks rather than replacing them.

What VLMsAreBlind Measures

VLMsAreBlind evaluates seven simple visual tasks: counting line intersections, determining whether two circles overlap, identifying which letter in a word is circled, counting circles in an Olympic-logo-style image, counting nested squares, counting rows and columns in a grid, and following color-coded paths on a subway-map-style diagram. Every task is designed so a human can answer correctly with near-100% accuracy at a glance.

Despite this, the paper found four state-of-the-art VLMs (including GPT-4o and Gemini 1.5 Pro) averaged only 58% accuracy across tasks, with the best model (Claude 3.5 Sonnet) reaching just under 78% — far short of human-level performance. Linear-probing experiments showed the underlying vision encoders actually do capture the correct visual information; the failure occurs when the language model decodes that information into a verbal answer.

The authors argue this reveals a structural weakness in how current VLM architectures translate detailed visual features into language, rather than a data or training-scale problem alone — a finding directly relevant to any production application relying on precise spatial reasoning.

Benchmark Specifications

How VLMsAreBlind Is Scored

Each of the 7 tasks is scored as a simple accuracy percentage (fraction of instances answered correctly), then averaged across tasks to produce an overall score. Because the correct answer is deterministic and unambiguous, there is no LLM-judge subjectivity involved.

State-of-the-Art Results

Scores fluctuate with model version and image resolution/line-width settings. See the official repository for current task-by-task standings.

VLMsAreBlind on Benchgen

No Benchgen results yet — be the first to run VLMsAreBlind.

VLMsAreBlind vs Other Benchmarks

VLMsAreBlind is narrower and harder-edged than general perception suites like MME, isolating a specific class of spatial-reasoning failure that broader benchmarks tend to average away.

Run VLMsAreBlind on Your Model

Benchgen lets teams run the BlindTest suite against their own model versions on demand, tracking whether spatial-reasoning regressions creep in across releases — a failure mode that's invisible in aggregate multimodal scores but can break real UI-automation or diagram-reading use cases.

Related Models & Benchmarks

Explore related multimodal benchmarks like MME and POPE to compare broad perception and hallucination evaluation with VLMsAreBlind's narrow spatial-acuity focus.

Frequently Asked Questions