# MuirBench ![Category](https://img.shields.io/badge/Category-Reasoning-1f6feb) ![Metric](https://img.shields.io/badge/Metric-%25%20Accuracy-2da44e) ![Saturation](https://img.shields.io/badge/Saturation-Low-d29922) ![Created](https://img.shields.io/badge/Created-Jun%202024-555) [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/abs/2406.09411) [![GitHub](https://img.shields.io/badge/GitHub-muirbench%2FMuirBench-181717?logo=github&logoColor=white)](https://github.com/muirbench/MuirBench) [![Dataset](https://img.shields.io/badge/Dataset-HuggingFace-ff9d00?logo=huggingface&logoColor=white)](https://huggingface.co/datasets/MUIRBENCH/MUIRBENCH) > **Quick answer:** MuirBench (Multi-Image Understanding and Reasoning Benchmark) tests a model's ability to reason across multiple images spanning 12 task types and 10 multi-image relations (e.g., diff-spotting, ordering, complementary views), and pairs each question with an "unanswerable" variant to check for shortcut guessing. LFM2.5-VL-3B scores 58.3% as of August 2026. ## At a Glance **What it tests:** Reasoning that requires integrating information across multiple images — comparison, ordering, complementary-view synthesis, and difference-spotting — rather than answering from a single image. **Why it matters:** Real-world multimodal use cases (comparing product photos, reasoning across video frames, cross-referencing document pages) require multi-image reasoning, which single-image benchmarks don't test. **Known limitations:** As a relatively new benchmark (2024), independent replication and long-term saturation trends are less established than for older single-image VQA benchmarks. ## What MuirBench Measures MuirBench was built to isolate multi-image reasoning as a distinct capability from single-image visual question answering. It spans 12 task types (including comparison, difference-spotting, ordering, complementary reasoning, and multi-view synthesis) across 10 categories of image relations. To guard against models exploiting single-image shortcuts, each question is paired with an "unanswerable" variant — a near-identical image set for which the correct answer is that the question cannot be answered — requiring genuine cross-image reasoning rather than pattern matching. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Reasoning (multi-image) | | Metric | % accuracy | | Saturation | Low | | Created by | Wang et al. | | Source paper | [Wang et al. 2024](https://arxiv.org/abs/2406.09411) | | GitHub | [muirbench/MuirBench](https://github.com/muirbench/MuirBench) | | Dataset | [MUIRBENCH/MUIRBENCH on HuggingFace](https://huggingface.co/datasets/MUIRBENCH/MUIRBENCH) | ## How MuirBench Is Scored Models answer multiple-choice questions that require reasoning across a set of images; score is the percentage of questions (including unanswerable-pair variants) answered correctly. ## State-of-the-Art Results | Rank | Model | Score | Source | Date | |---|---|---|---|---| | 1 | [LFM2.5-VL-3B](/models/liquid-ai/lfm2-5-vl-3b) | 58.3% | [LFM2.5-VL-3B launch blog](https://www.liquid.ai/blog/lfm2-5-vl-3b) | 2026-08 | *Score sourced from Liquid AI's LFM2.5-VL-3B announcement, August 2026.* ## MuirBench on Benchgen > No Benchgen results yet — [be the first to run MuirBench](/benchmarks/wang-et-al/muirbench). ## MuirBench vs Other Benchmarks | Benchmark | What it tests | Saturation | |---|---|---| | **MuirBench** | Multi-image reasoning across 12 task types | Low | | [BLINK](/benchmarks/fu-et-al/blink) | Core visual perception (mostly single/paired-image) | Low | ## Run MuirBench on Your Model Benchgen lets you evaluate your own vision-language model's multi-image reasoning and track it across model versions. ## Frequently Asked Questions

What is MuirBench?

MuirBench is a multi-image understanding and reasoning benchmark spanning 12 task types and 10 image relations, with paired unanswerable questions to check for shortcut guessing.

What does a good score look like on MuirBench?

LFM2.5-VL-3B reports 58.3% as of August 2026; larger frontier models generally score higher, reflecting the difficulty of genuine cross-image reasoning.

Who created MuirBench?

MuirBench was created by Wang et al.; see the [original paper](https://arxiv.org/abs/2406.09411).

How does MuirBench differ from BLINK?

MuirBench focuses specifically on reasoning across multiple images (comparison, ordering, multi-view synthesis), while BLINK covers a broader set of core visual perception tasks that are mostly single- or paired-image.

MuirBench — Results

MuirBench — Evaluations