# MMBench ![Category](https://img.shields.io/badge/Category-Reasoning-1f6feb) ![Metric](https://img.shields.io/badge/Metric-%25%20Accuracy-2da44e) ![Saturation](https://img.shields.io/badge/Saturation-Medium-d29922) ![Created](https://img.shields.io/badge/Created-Jul%202023-555) [![Paper](https://img.shields.io/badge/Paper-arXiv%202307.06281-b31b1b)](https://arxiv.org/abs/2307.06281) [![GitHub](https://img.shields.io/badge/GitHub-open--compass%2FMMBench-181717?logo=github&logoColor=white)](https://github.com/open-compass/MMBench) [![Dataset](https://img.shields.io/badge/Dataset-HuggingFace-ff9d00?logo=huggingface&logoColor=white)](https://huggingface.co/datasets/lmms-lab/MMBench) > **Quick answer:** MMBench (Liu et al., 2023) is a multimodal benchmark that evaluates vision-language models across roughly 20 fine-grained ability dimensions — from object localization to social reasoning — using a "circular evaluation" multiple-choice protocol designed to reduce lucky guessing. North Micro Vision scores 68.7% on the DEV_EN_V11 split as of August 2026. ## At a Glance **What it tests:** Around 20 fine-grained perception and reasoning abilities (e.g. object localization, attribute recognition, spatial relationship, social reasoning) via multiple-choice visual questions. **Why it matters:** MMBench's circular-evaluation protocol — rotating answer choices and requiring a model to get every rotation of the same question right — makes it harder to game than single-pass multiple-choice benchmarks, giving a more reliable read on genuine visual understanding. **Known limitations:** As one of the earlier widely-adopted VLM benchmarks, top frontier models are approaching saturation on several of its easier ability dimensions, though it remains discriminative for small/compact models. ## What MMBench Measures MMBench decomposes "visual understanding" into roughly 20 fine-grained, hierarchically-organized ability dimensions rather than treating VQA as one monolithic skill. Questions are multi-image or single-image multiple choice, spanning coarse perception (image style, scene understanding), fine-grained perception (identity, attribute, and object localization), and reasoning (physical/social relations, structuralized image-text understanding). Its signature contribution is "CircularEval": each question is presented multiple times with the answer choices cyclically shuffled, and a model only receives credit if it answers every permutation correctly. This substantially reduces the chance a model gets credit from guessing or exploiting positional biases in multiple-choice options, making MMBench a more robust yardstick than a single-pass multiple-choice format. Because MMBench predates many of today's frontier vision-language models, well-tuned large models now score quite highly on its English dev split — but it remains a useful and widely-reported comparison point for small and mid-size vision-language models like North Micro Vision, where score gaps of several points are still meaningful. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Reasoning (general VQA) | | Metric | % accuracy (CircularEval multiple-choice) | | Saturation | Medium | | Created by | Liu et al. | | Source paper | [Liu et al. 2023](https://arxiv.org/abs/2307.06281) | | GitHub | [open-compass/MMBench](https://github.com/open-compass/MMBench) | | Dataset | [lmms-lab/MMBench on HuggingFace](https://huggingface.co/datasets/lmms-lab/MMBench) | ## How MMBench Is Scored Each question is scored correct only if the model answers correctly across all cyclic shufflings of its multiple-choice options (CircularEval). The overall score is the percentage of questions passed under this stricter criterion, commonly reported on the `DEV_EN_V11` (English development) split. ## State-of-the-Art Results | Rank | Model | Score | Source | Date | |---|---|---|---|---| | 1 | [North Micro Vision Instruct](/models/cohere-labs/north-micro-vision-instruct) | 68.7% | [North Micro Vision launch blog](https://huggingface.co/blog/CohereLabs/meet-north-micro-vision-instruct) | 2026-08 | *Score sourced from Cohere Labs' North Micro Vision Instruct launch announcement, August 2026, evaluated on the `DEV_EN_V11` split via VLMEvalKit.* ## MMBench on Benchgen > No Benchgen results yet — [be the first to run MMBench](/benchmarks/liu-et-al/mmbench). ## MMBench vs Other Benchmarks | Benchmark | What it tests | Saturation | |---|---|---| | **MMBench** | ~20 fine-grained VQA ability dimensions, CircularEval | Medium | | [MMStar](/benchmarks/chen-et-al/mmstar) | Vision-indispensable, curated general VQA | Low | | [GQA](/benchmarks/hudson-and-manning/gqa) | Compositional real-world scene reasoning | High | MMBench's CircularEval protocol makes it a stricter general-VQA test than a plain single-pass multiple-choice benchmark, while MMStar specifically filters out questions answerable without the image at all. ## Run MMBench on Your Model Benchgen lets you evaluate your own vision-language model's fine-grained perception and reasoning abilities under the CircularEval protocol and track them across model versions. ## Related Models & Benchmarks - Top performers: [North Micro Vision Instruct](/models/cohere-labs/north-micro-vision-instruct) - Related benchmarks: [MMStar](/benchmarks/chen-et-al/mmstar) · [GQA](/benchmarks/hudson-and-manning/gqa) ## Frequently Asked Questions

What is MMBench?

MMBench is a multimodal benchmark that tests vision-language models across roughly 20 fine-grained ability dimensions using a robust "CircularEval" multiple-choice protocol.

What does a good MMBench score look like?

North Micro Vision Instruct reports 68.7% on the DEV_EN_V11 split as of August 2026; larger frontier models typically score in the high 70s to 80s.

Who created MMBench?

MMBench was created by Liu et al.; see the [original paper](https://arxiv.org/abs/2307.06281).

Is MMBench saturated?

It's showing medium saturation — frontier models perform well on many dimensions, but it remains discriminative for small and mid-size vision-language models.

MMBench — Results

MMBench — Evaluations