PMC-VQA
1 phaseActive
227,000 VQA pairs across 149,000 medical images from PubMed Central figures, plus a manually verified, more challenging test set.
Overview
PMC-VQA
Quick answer: PMC-VQA is a large-scale medical visual question answering dataset containing 227,000 QA pairs across 149,000 images sourced from PubMed Central figures, built via a scalable generative pipeline and complemented by a smaller, manually verified test set that is significantly more challenging.
At a Glance
What it tests: Medical visual understanding across diverse imaging modalities and diseases, framed as a generative task (producing a free-form answer) rather than pure multiple-choice classification.
Why it matters: Medical VQA presents a significant opportunity to enhance diagnostic accuracy and healthcare delivery, but prior medical VQA datasets were relatively small; PMC-VQA's scalable construction pipeline made it possible to build a much larger, diverse training and evaluation resource covering multiple modalities and conditions.
Known limitations: Because the bulk of the dataset is automatically constructed from PubMed Central figure captions, some QA pairs may be lower quality than fully human-curated sets — which is why the authors released a separate, manually verified test split for more reliable evaluation.
What PMC-VQA Measures
PMC-VQA reframes medical VQA as a generation task that naturally follows human-machine interaction, proposing a generative-based model (MedVInT) that aligns visual information from a pretrained vision encoder with a large language model. The authors establish a scalable pipeline to construct the large-scale PMC-VQA dataset, containing 227,000 VQA pairs across 149,000 images covering various imaging modalities and diseases sourced from PubMed Central biomedical literature figures.
Recognizing that automatically constructed QA pairs can vary in quality, the authors also release a manually verified test set that is significantly more challenging, intended to better track genuine progress in generative medical VQA methods rather than allowing models to exploit artifacts of the automated construction pipeline. Models trained on PMC-VQA and then fine-tuned on established public benchmarks (VQA-RAD, SLAKE, Image-Clef-2019) significantly outperformed prior MedVQA models at the time of release.
PMC-VQA maintains a public leaderboard to track state-of-the-art approaches to both classification-style and generative medical VQA.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Multimodal (medical visual question answering) |
| Metric | Accuracy (%) for classification-style evaluation; BLEU/other NLG metrics for free-form generative answers |
| Number of tasks | 227,000 VQA pairs across 149,000 images, plus a smaller manually verified test set |
| Saturation | Medium — the automatically constructed set shows higher scores than the harder, manually verified test set |
| Created by | Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, Weidi Xie |
| Source paper | Zhang et al. 2023 |
| Code | GitHub |
| Dataset | Hugging Face |
How PMC-VQA Is Scored
Multiple-choice-style questions are scored as accuracy against the correct option; free-form generative answers are scored with standard NLG metrics (BLEU and similar) against reference answers. The manually verified test set is scored identically but represents a meaningfully harder evaluation than the automatically constructed training/test data.
State-of-the-Art Results
Scores differ substantially between the original automatically-constructed test set and the harder, manually verified test set. See the PapersWithCode leaderboard for current standings.
PMC-VQA on Benchgen
No Benchgen results yet — be the first to run PMC-VQA.
PMC-VQA vs Other Benchmarks
| Benchmark | What it tests | Tasks | Saturation |
|---|---|---|---|
| PMC-VQA | Generative medical VQA from literature figures | 227,000 | medium |
| MedXpertQA | Expert-level medical reasoning, Text + MM subsets | 4,460 | low |
| MIMIC-CXR | Chest radiograph classification & report generation | 377,110 | medium |
PMC-VQA's literature-figure-derived breadth across imaging modalities complements MIMIC-CXR's clinical chest x-ray focus and MedXpertQA's specialty-board-level reasoning difficulty.
Run PMC-VQA on Your Model
Benchgen lets teams evaluate their own medical VQA models against both the standard and manually verified PMC-VQA test sets on demand, tracking generative accuracy over time to catch regressions before they reach production clinical tools.
Related Models & Benchmarks
Explore related multimodal benchmarks like MedXpertQA and MIMIC-CXR to compare literature-derived medical VQA with expert-level reasoning and clinical imaging evaluation.