# MedXpertQA ![Category](https://img.shields.io/badge/Category-Multimodal-1f6feb) ![Metric](https://img.shields.io/badge/Metric-Accuracy-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-4%2C460-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-low-d29922) ![Created](https://img.shields.io/badge/Created-Jan%202025-555) [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/abs/2501.18362) [![GitHub](https://img.shields.io/badge/GitHub-Code-181717?logo=github&logoColor=white)](https://github.com/TsinghuaC3I/MedXpertQA) > **Quick answer:** MedXpertQA is a highly challenging benchmark for expert-level medical knowledge and reasoning, comprising 4,460 questions across 17 specialties and 11 body systems, with a multimodal (MM) subset that introduces genuine specialty-board exam questions featuring patient records, exam results, and diverse clinical images rather than simple image-caption QA pairs. ## At a Glance **What it tests:** Expert-level clinical reasoning and medical knowledge via both a text-only subset and a multimodal subset that requires interpreting rich clinical context (patient history, lab results, imaging) alongside diverse medical images. **Why it matters:** Existing medical benchmarks like MedQA have become insufficiently difficult for frontier models; MedXpertQA applies rigorous filtering, augmentation, and specialty board questions to restore genuine difficulty, and its MM subset is explicitly designed to be harder than typical medical VQA sets built from simple image captions. **Known limitations:** As a highly specialized medical benchmark, correct interpretation requires domain expertise to validate answer quality beyond simple accuracy, and the benchmark's reasoning-oriented subset specifically targets o1-like models, which may not be directly comparable to standard instruction-tuned models. ## What MedXpertQA Measures MedXpertQA includes 4,460 questions spanning 17 medical specialties and 11 body systems, split into a Text subset for pure text-based medical reasoning and an MM (multimodal) subset that introduces expert-level exam questions featuring diverse images and rich clinical information — including patient records and examination results — setting it apart from traditional medical multimodal benchmarks built from simple image-caption-derived QA pairs. To address insufficient difficulty in prior benchmarks, the authors apply rigorous filtering and augmentation, incorporate specialty board exam questions for improved clinical relevance, and perform data synthesis specifically to mitigate data leakage risk. Multiple rounds of expert review were conducted to ensure both accuracy and reliability of the question set. Because medicine connects deeply to real-world decision-making, MedXpertQA also provides a rich setting for assessing reasoning abilities beyond mathematics and code, and the authors developed a dedicated reasoning-oriented subset to facilitate evaluation of o1-like reasoning models specifically. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Multimodal (expert-level medical reasoning; Text and MM subsets) | | Metric | Accuracy (%) | | Number of tasks | 4,460 questions across 17 specialties and 11 body systems | | Saturation | Low — designed specifically to restore difficulty lost to benchmark saturation | | Created by | Yuxin Zuo, Shang Qu, Yifei Li, Zhangren Chen, Xuekai Zhu, and collaborators | | Source paper | [Zuo et al. 2025](https://arxiv.org/abs/2501.18362) (ICML 2025) | | Code | [GitHub](https://github.com/TsinghuaC3I/MedXpertQA) | | Dataset | [Hugging Face](https://huggingface.co/datasets/TsinghuaC3I/MedXpertQA) | ## How MedXpertQA Is Scored Both the Text and MM subsets are scored as multiple-choice/open-ended accuracy against expert-verified ground truth, reported separately for each subset (and by specialty/body-system where relevant), since text-only and multimodal reasoning represent distinct capability profiles. ## State-of-the-Art Results > Scores vary significantly between the Text and MM subsets and by model release. See the paper and [official repository](https://github.com/TsinghuaC3I/MedXpertQA) for current standings across 18 evaluated models. ## MedXpertQA on Benchgen > No Benchgen results yet — [be the first to run MedXpertQA](/benchmarks/zuo-et-al-tsinghua-university/medxpertqa). ## MedXpertQA vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **MedXpertQA** | Expert-level medical reasoning, Text + MM subsets | 4,460 | low | | [PMC-VQA](/benchmarks/zhang-et-al-shanghai-jiao-tong-university/pmc-vqa) | Medical visual question answering from literature figures | 227000 | medium | | [MIMIC-CXR](/benchmarks/johnson-et-al-mit/mimic-cxr) | Chest radiograph interpretation with free-text reports | 377110 | medium | MedXpertQA targets specialty-board-level clinical reasoning difficulty across all of medicine, complementing narrower imaging-specific benchmarks like MIMIC-CXR (chest radiographs) and PMC-VQA (literature-derived figures). ## Run MedXpertQA on Your Model Benchgen lets teams run both the Text and MM subsets of MedXpertQA against their own model versions on demand, tracking accuracy by specialty and body system to catch regressions in specific areas of clinical reasoning. ## Related Models & Benchmarks Explore related multimodal benchmarks like [PMC-VQA](/benchmarks/zhang-et-al-shanghai-jiao-tong-university/pmc-vqa) and [MIMIC-CXR](/benchmarks/johnson-et-al-mit/mimic-cxr) to compare specialty-board-level medical reasoning with imaging-focused medical VQA evaluation. ## Frequently Asked Questions

What is MedXpertQA?

MedXpertQA is a 4,460-question benchmark for expert-level medical knowledge and reasoning across 17 specialties, with Text and multimodal (MM) subsets — the MM subset uses real specialty-board exam questions with clinical images and patient records.

What does a good MedXpertQA score look like?

Because MedXpertQA was specifically constructed to restore difficulty lost to saturation on benchmarks like MedQA, strong models typically show a meaningful accuracy gap between the Text and MM subsets, with MM generally proving harder due to its richer clinical context requirements.

Who created MedXpertQA?

MedXpertQA was created by Yuxin Zuo and collaborators at Tsinghua University and Shanghai AI Laboratory, accepted at ICML 2025.

Rank	Model	Score
1	o1	44.67
2	gpt-4o	30.37
3	qvq-72b-preview	21.76
4	claude-3-5-sonnet	21.31
5	gemini-2-0-flash	20.57
6	gemini-1-5-pro	19.67
7	qwen2-5-vl-72b	17.96
8	gpt-4o-mini	17.84
9	qwen2-vl-72b	16.94

MedXpertQA — Results

MedXpertQA — Evaluations