# VisuLogic ![Category](https://img.shields.io/badge/Category-Multimodal-1f6feb) ![Metric](https://img.shields.io/badge/Metric-Accuracy-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-1%2C000-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-low-d29922) ![Created](https://img.shields.io/badge/Created-Apr%202025-555) [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/abs/2504.15279) [![Project Page](https://img.shields.io/badge/Project-Page-2da44e)](https://visulogic-benchmark.github.io/VisuLogic) > **Quick answer:** VisuLogic is a benchmark of 1,000 human-verified problems across six categories (e.g., quantitative shifts, spatial relations, attribute comparisons) designed to test genuine vision-centric reasoning in multimodal LLMs, deliberately blocking the language-based shortcuts that let models "reason" from a text description alone. ## At a Glance **What it tests:** Whether an MLLM can perform visual reasoning that genuinely requires interpreting the image — not reasoning that could be solved from a text caption or description of the image. **Why it matters:** Many prior "visual reasoning" evaluations quietly allow models to succeed via language-only shortcuts (e.g., inferring the answer from a text description rather than the image itself); VisuLogic's problems are specifically constructed to prevent this. **Known limitations:** With most leading MLLMs scoring below 30% (barely above the 25% random baseline) versus 51.4% for humans, VisuLogic exposes a large capability gap rather than finely differentiating among current frontier models. ## What VisuLogic Measures VisuLogic presents 1,000 human-verified visual reasoning problems spanning six categories — including quantitative shifts, spatial relations, and attribute comparisons — each requiring the model to detect patterns or relationships directly from visual content. Unlike many multimodal reasoning benchmarks, VisuLogic's problems cannot be solved via a text description or caption of the image; a model must genuinely parse and reason over pixel-level visual information. The paper's evaluation of leading MLLMs found that most models score below 30% accuracy — only slightly above the 25% random baseline for the four-option multiple-choice format — while humans reach 51.4%, revealing a stark gap in vision-centric reasoning compared to the strong results many models post on other multimodal benchmarks that permit language shortcuts. The authors also release a supplementary training dataset and a reinforcement-learning baseline to support future progress. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Multimodal (vision-centric abstract reasoning) | | Metric | Multiple-choice accuracy (%) | | Number of tasks | 1,000 human-verified problems across 6 reasoning categories | | Saturation | Low — most MLLMs score below 30%, far below the 51.4% human baseline | | Created by | Weiye Xu and 12 co-authors (Shanghai AI Laboratory, USTC, and collaborators) | | Source paper | [Xu et al. 2025](https://arxiv.org/abs/2504.15279) | | Code, data & baselines | [Project page](https://visulogic-benchmark.github.io/VisuLogic) | ## How VisuLogic Is Scored Each problem is a multiple-choice question (typically four options) requiring the model to identify a visual pattern, transformation, or relationship. Accuracy is the percentage of problems answered correctly, reported overall and broken down by the six reasoning categories. ## State-of-the-Art Results > Scores fluctuate by model release cadence and evaluation harness (prompt template, use of chain-of-thought). See the [project page](https://visulogic-benchmark.github.io/VisuLogic) for current standings. ## VisuLogic on Benchgen > No Benchgen results yet — [be the first to run VisuLogic](/benchmarks/weiye-xu-et-al-shanghai-ai-laboratory-ustc/visulogic). ## VisuLogic vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **VisuLogic** | Vision-centric abstract reasoning, blocking language shortcuts | 1,000 | low | | [VLMsAreBlind](/benchmarks/vlmsareblind) | Low-level visual perception primitives | — | low | | [ZebraLogic](/benchmarks/reasoning-team/zebralogic) | Text-based logic grid puzzles | — | medium | VisuLogic's vision-centric focus complements VLMsAreBlind's low-level perception tests and contrasts with text-only logic benchmarks like ZebraLogic, isolating how much of "reasoning" performance is actually driven by visual understanding versus language priors. ## Run VisuLogic on Your Model Benchgen lets teams run the full VisuLogic suite against their own model versions on demand, tracking accuracy across all six reasoning categories to gauge genuine vision-centric reasoning ability. ## Related Models & Benchmarks Explore related visual reasoning benchmarks like [VLMsAreBlind](/benchmarks/vlmsareblind) and text-based logic benchmarks like [ZebraLogic](/benchmarks/reasoning-team/zebralogic) to compare vision-centric and language-centric reasoning evaluation approaches. ## Frequently Asked Questions

What is VisuLogic?

VisuLogic is a benchmark of 1,000 human-verified problems across six categories designed to test genuine vision-centric reasoning in multimodal LLMs, specifically blocking language-based shortcuts.

What does a good VisuLogic score look like?

Most leading MLLMs score below 30% accuracy, barely above the 25% random baseline, while humans achieve 51.4% — so any score meaningfully above 30-35% would represent a notable result.

Who created VisuLogic?

VisuLogic was created by Weiye Xu and a team of collaborators primarily at Shanghai AI Laboratory and USTC, published in April 2025.

VisuLogic — Phases

VisuLogic