# VCR_en_easy ![Category](https://img.shields.io/badge/Category-Reasoning-1f6feb) ![Metric](https://img.shields.io/badge/Metric-Exact%20Match-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-2.11M%20(English)-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-low-d29922) ![Created](https://img.shields.io/badge/Created-Jun%202024-555) [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/abs/2406.06462) [![GitHub](https://img.shields.io/badge/GitHub-tianyu--z%2FVCR-181717?logo=github&logoColor=white)](https://github.com/tianyu-z/VCR) > **Quick answer:** VCR_en_easy is the English, easy-difficulty split of the Visual Caption Restoration (VCR-Wiki) benchmark, which challenges vision-language models to restore partially occluded text embedded in an image using pixel-level visual hints combined with surrounding image and text context — a task that requires genuinely fusing vision and language rather than relying on OCR alone. ## At a Glance **What it tests:** Whether a vision-language model can reconstruct masked text in an image by combining subtle visual cues from the tiny exposed pixel regions of the occluded text with broader visual and linguistic context, rather than simply reading unobstructed text via OCR. **Why it matters:** Real-world documents, screenshots, and UI images frequently contain partially obscured or degraded text; VCR isolates whether a model can genuinely integrate vision and language to recover it, a capability current OCR-plus-language-model pipelines are not designed to handle. **Known limitations:** Even fine-tuning models directly on the VCR-Wiki dataset does not lead to notable improvements per the original paper, suggesting the task probes a more fundamental architectural gap in vision-language fusion rather than a simple data-coverage gap. ## What VCR_en_easy Measures VCR-Wiki is constructed via a synthetic-image generation pipeline that takes image-caption pairs from Wikipedia and overlays each caption onto its corresponding image with an adjustable visibility level, deliberately masking most of each word's pixels while leaving small exposed fragments. The full dataset comprises 2.11 million English and 346,000 Chinese image-caption entities, each released in both "easy" and "hard" difficulty variants that control how much of the caption text remains visible. **VCR_en_easy** specifically refers to the English-language, easy-difficulty configuration, where a larger fraction of each masked word's pixels remains visible, making the restoration task comparatively more tractable than the hard variant. Crucially, because the caption text is split across small, disconnected pixel fragments rather than presented as clean OCR-able text, a model cannot solve VCR by extracting text and then reasoning purely in the language domain — it must fuse the sparse visual pixel hints with the surrounding image content and any available textual context to reconstruct the full caption correctly. The paper's central finding is that current vision-language models significantly lag behind human performance on this task, and that directly fine-tuning models on VCR-Wiki does not meaningfully close the gap, pointing to a deeper limitation in how current VLMs integrate text-in-image and free-form visual understanding. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Reasoning / pixel-level vision-language text restoration | | Metric | Exact match / accuracy on the restored caption text | | Number of tasks | 2.11M English entities (VCR-Wiki total); VCR_en_easy is the English, easy-difficulty split | | Saturation | Low — current VLMs significantly lag human performance, even after fine-tuning | | Created by | Tianyu Zhang, Suyuchen Wang, Lu Li, Ge Zhang, Perouz Taslakian, Sai Rajeswar, Jie Fu, Bang Liu, Yoshua Bengio (Mila – Quebec AI Institute) | | Source paper | [Zhang et al. 2025](https://arxiv.org/abs/2406.06462) (ICLR 2025) | | GitHub | [tianyu-z/VCR](https://github.com/tianyu-z/VCR) | ## How VCR_en_easy Is Scored A model is shown the partially-masked image and asked to output the full, exact original caption text; scoring is based on exact-match or near-exact string accuracy between the model's restored caption and the ground-truth original caption, typically also reported with a more lenient edit-distance-based partial-credit metric to distinguish near-misses from wholesale failures. ## State-of-the-Art Results > Current vision-language models significantly lag behind human performance on VCR-Wiki tasks, including the easy-difficulty splits. See the [official GitHub repository](https://github.com/tianyu-z/VCR) and paper for the full per-model breakdown across easy/hard and English/Chinese configurations. ## VCR_en_easy on Benchgen > No Benchgen results yet — [be the first to run VCR_en_easy](/benchmarks/tianyu-zhang-suyuchen-wang-lu-li-ge-zhang-perouz-taslakian-sai-rajeswar-jie-fu-bang-liu-yoshua-bengio-mila-quebec-ai-institute/vcr_en_easy). ## VCR_en_easy vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **VCR_en_easy** | Pixel-level text restoration in images (English, easy) | 2.11M | low | | [HallusionBench](/benchmarks/hallusionbench) | Visual hallucination resistance | — | medium | | [FRAMES](/benchmarks/satyapriya-krishna-et-al-google-deepmind-harvard-university/frames) | Multi-hop retrieval-augmented reasoning | — | low | VCR_en_easy is distinguished from most vision-language benchmarks by requiring genuine pixel-level visual-textual fusion — reconstructing text from sparse exposed pixel fragments — rather than testing visual question answering or hallucination resistance over already-legible content. ## Run VCR_en_easy on Your Model Benchgen lets teams evaluate their own vision-language model's text-restoration accuracy on the English easy-difficulty VCR-Wiki split, benchmarking against the human-performance gap the original paper identified. ## Frequently Asked Questions

What is VCR_en_easy?

VCR_en_easy is the English, easy-difficulty split of the VCR-Wiki benchmark, testing whether vision-language models can restore partially occluded text in images using pixel-level and contextual cues.

What does a good VCR_en_easy score look like?

Current vision-language models significantly lag behind human performance even on the easier difficulty split, so accuracy approaching human-level restoration represents strong, currently rare performance.

Who created VCR-Wiki?

VCR-Wiki was created by Tianyu Zhang, Suyuchen Wang, and collaborators including Yoshua Bengio at Mila – Quebec AI Institute, accepted at ICLR 2025.

Rank	Model	Score
1	gpt-4o	91.55
2	qwen2-vl-72b	91.3
3	qwen2-vl-7b	89.7
4	internvl2-40b	84.67
5	gpt-4o-mini	83.6
6	gpt-4-turbo	78.74
7	qwen-vl-max	76.8
8	claude-3-5-sonnet	63.85
9	gemini-1-5-pro	62.73
10	claude-3-opus	62

VCR_en_easy — Results

VCR_en_easy — Evaluations