# MMStar ![Category](https://img.shields.io/badge/Category-Reasoning-1f6feb) ![Metric](https://img.shields.io/badge/Metric-%25%20Accuracy-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-1500-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-Low-d29922) ![Created](https://img.shields.io/badge/Created-Mar%202024-555) [![Paper](https://img.shields.io/badge/Paper-arXiv%202403.20330-b31b1b)](https://arxiv.org/abs/2403.20330) [![GitHub](https://img.shields.io/badge/GitHub-MMStar--Benchmark%2FMMStar-181717?logo=github&logoColor=white)](https://github.com/MMStar-Benchmark/MMStar) [![Dataset](https://img.shields.io/badge/Dataset-HuggingFace-ff9d00?logo=huggingface&logoColor=white)](https://huggingface.co/datasets/Lin-Chen/MMStar) > **Quick answer:** MMStar (Chen et al., 2024) is a curated, "vision-indispensable" multimodal benchmark of 1,500 samples, hand-selected so every question genuinely requires the image — filtering out the data leakage and text-answerable questions that inflate scores on earlier VQA benchmarks. North Micro Vision scores 51.8% as of August 2026. ## At a Glance **What it tests:** Six core capabilities — coarse perception, fine-grained perception, instance reasoning, logical reasoning, science & technology, and math — using samples verified to require visual input. **Why it matters:** The paper behind MMStar showed many existing VLM benchmarks contain questions answerable by strong LLMs without ever seeing the image, plus training-data leakage; MMStar was built to eliminate both issues, giving a purer read on multimodal capability. **Known limitations:** At only 1,500 samples, per-category scores can be noisier than larger benchmarks, though the overall score remains a reliable comparative signal. ## What MMStar Measures MMStar's authors ran a systematic audit of popular multimodal benchmarks and found that a meaningful fraction of questions could be answered correctly by text-only LLMs with no image at all, or were already present in models' training data. MMStar was built by human-curating 1,500 samples specifically to eliminate both failure modes — every question requires genuine visual content to answer, and samples were checked against known training corpora to minimize leakage. The benchmark organizes questions into six core capability dimensions: coarse perception, fine-grained perception, instance reasoning, logical reasoning, science & technology, and mathematics — giving a capability-level breakdown alongside the overall score. Because it was explicitly designed against leakage and text-only shortcuts, MMStar tends to produce lower, more discriminative scores than older general-VQA benchmarks for the same model, making it a useful "sanity check" alongside benchmarks like MMBench. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Reasoning (curated general VQA) | | Metric | % accuracy | | Number of tasks | 1,500 | | Saturation | Low | | Created by | Chen et al. | | Source paper | [Chen et al. 2024](https://arxiv.org/abs/2403.20330) | | GitHub | [MMStar-Benchmark/MMStar](https://github.com/MMStar-Benchmark/MMStar) | | Dataset | [Lin-Chen/MMStar on HuggingFace](https://huggingface.co/datasets/Lin-Chen/MMStar) | ## How MMStar Is Scored Each of the 1,500 multiple-choice/short-answer samples is scored for correctness; the overall score is the percentage answered correctly across all six capability categories, with per-category breakdowns also commonly reported. ## State-of-the-Art Results | Rank | Model | Score | Source | Date | |---|---|---|---|---| | 1 | [North Micro Vision Instruct](/models/cohere-labs/north-micro-vision-instruct) | 51.8% | [North Micro Vision launch blog](https://huggingface.co/blog/CohereLabs/meet-north-micro-vision-instruct) | 2026-08 | *Score sourced from Cohere Labs' North Micro Vision Instruct launch announcement, August 2026.* ## MMStar on Benchgen > No Benchgen results yet — [be the first to run MMStar](/benchmarks/chen-et-al/mmstar). ## MMStar vs Other Benchmarks | Benchmark | What it tests | Saturation | |---|---|---| | **MMStar** | Curated, vision-indispensable general VQA | Low | | [MMBench](/benchmarks/liu-et-al/mmbench) | ~20 fine-grained VQA ability dimensions, CircularEval | Medium | | [GQA](/benchmarks/hudson-and-manning/gqa) | Compositional real-world scene reasoning | High | MMStar is the stricter, leakage-resistant complement to MMBench — use it when you need higher confidence that a score reflects genuine visual understanding rather than text-only shortcuts or memorization. ## Run MMStar on Your Model Benchgen lets you evaluate your own vision-language model against MMStar's curated, vision-indispensable sample set and track results across model versions. ## Related Models & Benchmarks - Top performers: [North Micro Vision Instruct](/models/cohere-labs/north-micro-vision-instruct) - Related benchmarks: [MMBench](/benchmarks/liu-et-al/mmbench) · [GQA](/benchmarks/hudson-and-manning/gqa) ## Frequently Asked Questions

What is MMStar?

MMStar is a curated multimodal benchmark of 1,500 samples specifically vetted so every question requires the image to answer, filtering out text-answerable and leaked questions found in other VQA benchmarks.

What does a good MMStar score look like?

North Micro Vision Instruct reports 51.8% as of August 2026; because MMStar filters out "easy" shortcut questions, scores are generally lower than on less-curated VQA benchmarks.

Who created MMStar?

MMStar was created by Chen et al.; see the [original paper](https://arxiv.org/abs/2403.20330).

Is MMStar saturated?

No — MMStar remains a low-saturation, discriminative benchmark by design, since it specifically excludes questions that are easy shortcuts for models.

MMStar — Results

MMStar — Evaluations