MMStar

MMStar

Quick answer: MMStar (Chen et al., 2024) is a curated, "vision-indispensable" multimodal benchmark of 1,500 samples, hand-selected so every question genuinely requires the image — filtering out the data leakage and text-answerable questions that inflate scores on earlier VQA benchmarks. North Micro Vision scores 51.8% as of August 2026.

At a Glance

What it tests: Six core capabilities — coarse perception, fine-grained perception, instance reasoning, logical reasoning, science & technology, and math — using samples verified to require visual input.

Why it matters: The paper behind MMStar showed many existing VLM benchmarks contain questions answerable by strong LLMs without ever seeing the image, plus training-data leakage; MMStar was built to eliminate both issues, giving a purer read on multimodal capability.

Known limitations: At only 1,500 samples, per-category scores can be noisier than larger benchmarks, though the overall score remains a reliable comparative signal.

What MMStar Measures

MMStar's authors ran a systematic audit of popular multimodal benchmarks and found that a meaningful fraction of questions could be answered correctly by text-only LLMs with no image at all, or were already present in models' training data. MMStar was built by human-curating 1,500 samples specifically to eliminate both failure modes — every question requires genuine visual content to answer, and samples were checked against known training corpora to minimize leakage.

The benchmark organizes questions into six core capability dimensions: coarse perception, fine-grained perception, instance reasoning, logical reasoning, science & technology, and mathematics — giving a capability-level breakdown alongside the overall score.

Because it was explicitly designed against leakage and text-only shortcuts, MMStar tends to produce lower, more discriminative scores than older general-VQA benchmarks for the same model, making it a useful "sanity check" alongside benchmarks like MMBench.

Benchmark Specifications

How MMStar Is Scored

Each of the 1,500 multiple-choice/short-answer samples is scored for correctness; the overall score is the percentage answered correctly across all six capability categories, with per-category breakdowns also commonly reported.

State-of-the-Art Results

Score sourced from Cohere Labs' North Micro Vision Instruct launch announcement, August 2026.

MMStar on Benchgen

No Benchgen results yet — be the first to run MMStar.

MMStar vs Other Benchmarks

MMStar is the stricter, leakage-resistant complement to MMBench — use it when you need higher confidence that a score reflects genuine visual understanding rather than text-only shortcuts or memorization.

Run MMStar on Your Model

Benchgen lets you evaluate your own vision-language model against MMStar's curated, vision-indispensable sample set and track results across model versions.

Related Models & Benchmarks

• Top performers: North Micro Vision Instruct
• Related benchmarks: MMBench · GQA

Frequently Asked Questions