VCR_en_easy

VCR_en_easy

Quick answer: VCR_en_easy is the English, easy-difficulty split of the Visual Caption Restoration (VCR-Wiki) benchmark, which challenges vision-language models to restore partially occluded text embedded in an image using pixel-level visual hints combined with surrounding image and text context — a task that requires genuinely fusing vision and language rather than relying on OCR alone.

At a Glance

What it tests: Whether a vision-language model can reconstruct masked text in an image by combining subtle visual cues from the tiny exposed pixel regions of the occluded text with broader visual and linguistic context, rather than simply reading unobstructed text via OCR.

Why it matters: Real-world documents, screenshots, and UI images frequently contain partially obscured or degraded text; VCR isolates whether a model can genuinely integrate vision and language to recover it, a capability current OCR-plus-language-model pipelines are not designed to handle.

Known limitations: Even fine-tuning models directly on the VCR-Wiki dataset does not lead to notable improvements per the original paper, suggesting the task probes a more fundamental architectural gap in vision-language fusion rather than a simple data-coverage gap.

What VCR_en_easy Measures

VCR-Wiki is constructed via a synthetic-image generation pipeline that takes image-caption pairs from Wikipedia and overlays each caption onto its corresponding image with an adjustable visibility level, deliberately masking most of each word's pixels while leaving small exposed fragments. The full dataset comprises 2.11 million English and 346,000 Chinese image-caption entities, each released in both "easy" and "hard" difficulty variants that control how much of the caption text remains visible. VCR_en_easy specifically refers to the English-language, easy-difficulty configuration, where a larger fraction of each masked word's pixels remains visible, making the restoration task comparatively more tractable than the hard variant.

Crucially, because the caption text is split across small, disconnected pixel fragments rather than presented as clean OCR-able text, a model cannot solve VCR by extracting text and then reasoning purely in the language domain — it must fuse the sparse visual pixel hints with the surrounding image content and any available textual context to reconstruct the full caption correctly. The paper's central finding is that current vision-language models significantly lag behind human performance on this task, and that directly fine-tuning models on VCR-Wiki does not meaningfully close the gap, pointing to a deeper limitation in how current VLMs integrate text-in-image and free-form visual understanding.

Benchmark Specifications

How VCR_en_easy Is Scored

A model is shown the partially-masked image and asked to output the full, exact original caption text; scoring is based on exact-match or near-exact string accuracy between the model's restored caption and the ground-truth original caption, typically also reported with a more lenient edit-distance-based partial-credit metric to distinguish near-misses from wholesale failures.

State-of-the-Art Results

Current vision-language models significantly lag behind human performance on VCR-Wiki tasks, including the easy-difficulty splits. See the official GitHub repository and paper for the full per-model breakdown across easy/hard and English/Chinese configurations.

VCR_en_easy on Benchgen

No Benchgen results yet — be the first to run VCR_en_easy.

VCR_en_easy vs Other Benchmarks

VCR_en_easy is distinguished from most vision-language benchmarks by requiring genuine pixel-level visual-textual fusion — reconstructing text from sparse exposed pixel fragments — rather than testing visual question answering or hallucination resistance over already-legible content.

Run VCR_en_easy on Your Model

Benchgen lets teams evaluate their own vision-language model's text-restoration accuracy on the English easy-difficulty VCR-Wiki split, benchmarking against the human-performance gap the original paper identified.

Frequently Asked Questions