Benchgen

MuirBench — Results

RankModelScore
1lfm2-5-vl-3b58.3
M

MuirBench

1 phaseActive

Multi-image understanding benchmark spanning 12 task types and 10 multi-image relations, with unanswerable-pair robustness checks.

Overview

MuirBench

Category Metric Saturation Created

Paper GitHub Dataset

Quick answer: MuirBench (Multi-Image Understanding and Reasoning Benchmark) tests a model's ability to reason across multiple images spanning 12 task types and 10 multi-image relations (e.g., diff-spotting, ordering, complementary views), and pairs each question with an "unanswerable" variant to check for shortcut guessing. LFM2.5-VL-3B scores 58.3% as of August 2026.

At a Glance

What it tests: Reasoning that requires integrating information across multiple images — comparison, ordering, complementary-view synthesis, and difference-spotting — rather than answering from a single image.

Why it matters: Real-world multimodal use cases (comparing product photos, reasoning across video frames, cross-referencing document pages) require multi-image reasoning, which single-image benchmarks don't test.

Known limitations: As a relatively new benchmark (2024), independent replication and long-term saturation trends are less established than for older single-image VQA benchmarks.

What MuirBench Measures

MuirBench was built to isolate multi-image reasoning as a distinct capability from single-image visual question answering. It spans 12 task types (including comparison, difference-spotting, ordering, complementary reasoning, and multi-view synthesis) across 10 categories of image relations. To guard against models exploiting single-image shortcuts, each question is paired with an "unanswerable" variant — a near-identical image set for which the correct answer is that the question cannot be answered — requiring genuine cross-image reasoning rather than pattern matching.

Benchmark Specifications

FieldValue
Task categoryReasoning (multi-image)
Metric% accuracy
SaturationLow
Created byWang et al.
Source paperWang et al. 2024
GitHubmuirbench/MuirBench
DatasetMUIRBENCH/MUIRBENCH on HuggingFace

How MuirBench Is Scored

Models answer multiple-choice questions that require reasoning across a set of images; score is the percentage of questions (including unanswerable-pair variants) answered correctly.

State-of-the-Art Results

RankModelScoreSourceDate
1LFM2.5-VL-3B58.3%LFM2.5-VL-3B launch blog2026-08

Score sourced from Liquid AI's LFM2.5-VL-3B announcement, August 2026.

MuirBench on Benchgen

No Benchgen results yet — be the first to run MuirBench.

MuirBench vs Other Benchmarks

BenchmarkWhat it testsSaturation
MuirBenchMulti-image reasoning across 12 task typesLow
BLINKCore visual perception (mostly single/paired-image)Low

Run MuirBench on Your Model

Benchgen lets you evaluate your own vision-language model's multi-image reasoning and track it across model versions.

Frequently Asked Questions

What is MuirBench? MuirBench is a multi-image understanding and reasoning benchmark spanning 12 task types and 10 image relations, with paired unanswerable questions to check for shortcut guessing.
What does a good score look like on MuirBench? LFM2.5-VL-3B reports 58.3% as of August 2026; larger frontier models generally score higher, reflecting the difficulty of genuine cross-image reasoning.
Who created MuirBench? MuirBench was created by Wang et al.; see the original paper.
How does MuirBench differ from BLINK? MuirBench focuses specifically on reasoning across multiple images (comparison, ordering, multi-view synthesis), while BLINK covers a broader set of core visual perception tasks that are mostly single- or paired-image.