MMBench

MMBench

Quick answer: MMBench (Liu et al., 2023) is a multimodal benchmark that evaluates vision-language models across roughly 20 fine-grained ability dimensions — from object localization to social reasoning — using a "circular evaluation" multiple-choice protocol designed to reduce lucky guessing. North Micro Vision scores 68.7% on the DEV_EN_V11 split as of August 2026.

At a Glance

What it tests: Around 20 fine-grained perception and reasoning abilities (e.g. object localization, attribute recognition, spatial relationship, social reasoning) via multiple-choice visual questions.

Why it matters: MMBench's circular-evaluation protocol — rotating answer choices and requiring a model to get every rotation of the same question right — makes it harder to game than single-pass multiple-choice benchmarks, giving a more reliable read on genuine visual understanding.

Known limitations: As one of the earlier widely-adopted VLM benchmarks, top frontier models are approaching saturation on several of its easier ability dimensions, though it remains discriminative for small/compact models.

What MMBench Measures

MMBench decomposes "visual understanding" into roughly 20 fine-grained, hierarchically-organized ability dimensions rather than treating VQA as one monolithic skill. Questions are multi-image or single-image multiple choice, spanning coarse perception (image style, scene understanding), fine-grained perception (identity, attribute, and object localization), and reasoning (physical/social relations, structuralized image-text understanding).

Its signature contribution is "CircularEval": each question is presented multiple times with the answer choices cyclically shuffled, and a model only receives credit if it answers every permutation correctly. This substantially reduces the chance a model gets credit from guessing or exploiting positional biases in multiple-choice options, making MMBench a more robust yardstick than a single-pass multiple-choice format.

Because MMBench predates many of today's frontier vision-language models, well-tuned large models now score quite highly on its English dev split — but it remains a useful and widely-reported comparison point for small and mid-size vision-language models like North Micro Vision, where score gaps of several points are still meaningful.

Benchmark Specifications

How MMBench Is Scored

Each question is scored correct only if the model answers correctly across all cyclic shufflings of its multiple-choice options (CircularEval). The overall score is the percentage of questions passed under this stricter criterion, commonly reported on the DEV_EN_V11 (English development) split.

State-of-the-Art Results

Score sourced from Cohere Labs' North Micro Vision Instruct launch announcement, August 2026, evaluated on the DEV_EN_V11 split via VLMEvalKit.

MMBench on Benchgen

No Benchgen results yet — be the first to run MMBench.

MMBench vs Other Benchmarks

MMBench's CircularEval protocol makes it a stricter general-VQA test than a plain single-pass multiple-choice benchmark, while MMStar specifically filters out questions answerable without the image at all.

Run MMBench on Your Model

Benchgen lets you evaluate your own vision-language model's fine-grained perception and reasoning abilities under the CircularEval protocol and track them across model versions.

Related Models & Benchmarks

• Top performers: North Micro Vision Instruct
• Related benchmarks: MMStar · GQA

Frequently Asked Questions