# CountBench ![Category](https://img.shields.io/badge/Category-Reasoning-1f6feb) ![Metric](https://img.shields.io/badge/Metric-%25%20Accuracy-2da44e) ![Saturation](https://img.shields.io/badge/Saturation-Medium-d29922) ![Created](https://img.shields.io/badge/Created-Feb%202023-555) [![Paper](https://img.shields.io/badge/Paper-arXiv%202302.12066-b31b1b)](https://arxiv.org/abs/2302.12066) [![Project](https://img.shields.io/badge/Project-teaching--clip--to--count-0969da)](https://teaching-clip-to-count.github.io/) > **Quick answer:** CountBench (Paiss et al., 2023) is an object-counting benchmark built from real-world images that each contain between 2 and 10 instances of a single salient object type, testing whether a vision-language model can produce a precise count rather than a vague quantity estimate. North Micro Vision scores 72.5% as of August 2026. ## At a Glance **What it tests:** Exact object counting in natural images — given a photo containing 2 to 10 instances of one object type, the model must output the precise integer count. **Why it matters:** Counting is a deceptively hard capability for vision-language models — many produce plausible-sounding but numerically wrong answers, especially as object count or visual clutter increases; CountBench isolates this specific failure mode from broader visual reasoning. **Known limitations:** Coverage is limited to the 2-10 object-count range and to images with one clearly salient countable object type, so it doesn't test counting under heavy occlusion or in scenes with many overlapping object classes. ## What CountBench Measures CountBench was introduced as part of research into teaching vision-language models like CLIP to count, using real-world images (rather than synthetic renders) that each depict between 2 and 10 instances of a single, clearly identifiable object type. The task is simple to state — output the number of instances of the named object — but proved to be a genuine blind spot for many contemporary vision-language models, which tend to default to rough estimates or common "round" numbers rather than precisely counting. Because the images are drawn from natural photography rather than synthetic scenes, CountBench also implicitly tests robustness to real-world visual clutter, partial occlusion, and varying object scale, on top of the core counting skill. Grounding and counting are frequently grouped together as a single "spatial precision" capability area in modern VLM evaluation suites (alongside benchmarks like RefCOCO), since both require accurate, fine-grained visual localization rather than gist-level scene understanding. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Reasoning (visual counting) | | Metric | % accuracy (exact count match) | | Saturation | Medium | | Created by | Paiss et al. | | Source paper | [Paiss et al. 2023](https://arxiv.org/abs/2302.12066) | | Project page | [teaching-clip-to-count.github.io](https://teaching-clip-to-count.github.io/) | ## How CountBench Is Scored The model outputs an integer count for the named object in each image; the score is the percentage of images for which the predicted count exactly matches the ground-truth count (2 through 10 instances). ## State-of-the-Art Results | Rank | Model | Score | Source | Date | |---|---|---|---|---| | 1 | [North Micro Vision Instruct](/models/cohere-labs/north-micro-vision-instruct) | 72.5% | [North Micro Vision launch blog](https://huggingface.co/blog/CohereLabs/meet-north-micro-vision-instruct) | 2026-08 | *Score sourced from Cohere Labs' North Micro Vision Instruct launch announcement, August 2026.* ## CountBench on Benchgen > No Benchgen results yet — [be the first to run CountBench](/benchmarks/paiss-et-al/countbench). ## CountBench vs Other Benchmarks | Benchmark | What it tests | Saturation | |---|---|---| | **CountBench** | Exact object counting (2-10 instances) in real photos | Medium | | [RefCOCO](/benchmarks/kazemzadeh-et-al/refcoco) | Referring expression comprehension (object grounding) | Medium | | [GQA](/benchmarks/hudson-and-manning/gqa) | Compositional real-world scene reasoning | High | CountBench and RefCOCO are commonly grouped as a "grounding and counting" capability pair, since both require precise, fine-grained visual localization rather than gist-level scene understanding. ## Run CountBench on Your Model Benchgen lets you evaluate your own vision-language model's precise object-counting ability on real-world images and track results across model versions. ## Related Models & Benchmarks - Top performers: [North Micro Vision Instruct](/models/cohere-labs/north-micro-vision-instruct) - Related benchmarks: [RefCOCO](/benchmarks/kazemzadeh-et-al/refcoco) · [GQA](/benchmarks/hudson-and-manning/gqa) ## Frequently Asked Questions

What is CountBench?

CountBench is a benchmark testing whether vision-language models can precisely count instances of an object (2 to 10) in real-world photos.

What does a good CountBench score look like?

North Micro Vision Instruct reports 72.5% as of August 2026; scores tend to drop as the true object count increases toward the upper end of the 2-10 range.

Who created CountBench?

CountBench was introduced by Paiss et al.; see the [original paper](https://arxiv.org/abs/2302.12066).

Is CountBench saturated?

It shows medium saturation — precise counting remains a genuine weak spot for many vision-language models, even those strong on broader visual reasoning.

Rank	Model	Score
1	lfm3-vl-2b	93
2	lfm2-5-vl-3b	87.3
3	north-micro-vision-instruct	72.5

CountBench — Results

CountBench — Evaluations