MedXpertQA

MedXpertQA

Quick answer: MedXpertQA is a highly challenging benchmark for expert-level medical knowledge and reasoning, comprising 4,460 questions across 17 specialties and 11 body systems, with a multimodal (MM) subset that introduces genuine specialty-board exam questions featuring patient records, exam results, and diverse clinical images rather than simple image-caption QA pairs.

At a Glance

What it tests: Expert-level clinical reasoning and medical knowledge via both a text-only subset and a multimodal subset that requires interpreting rich clinical context (patient history, lab results, imaging) alongside diverse medical images.

Why it matters: Existing medical benchmarks like MedQA have become insufficiently difficult for frontier models; MedXpertQA applies rigorous filtering, augmentation, and specialty board questions to restore genuine difficulty, and its MM subset is explicitly designed to be harder than typical medical VQA sets built from simple image captions.

Known limitations: As a highly specialized medical benchmark, correct interpretation requires domain expertise to validate answer quality beyond simple accuracy, and the benchmark's reasoning-oriented subset specifically targets o1-like models, which may not be directly comparable to standard instruction-tuned models.

What MedXpertQA Measures

MedXpertQA includes 4,460 questions spanning 17 medical specialties and 11 body systems, split into a Text subset for pure text-based medical reasoning and an MM (multimodal) subset that introduces expert-level exam questions featuring diverse images and rich clinical information — including patient records and examination results — setting it apart from traditional medical multimodal benchmarks built from simple image-caption-derived QA pairs.

To address insufficient difficulty in prior benchmarks, the authors apply rigorous filtering and augmentation, incorporate specialty board exam questions for improved clinical relevance, and perform data synthesis specifically to mitigate data leakage risk. Multiple rounds of expert review were conducted to ensure both accuracy and reliability of the question set.

Because medicine connects deeply to real-world decision-making, MedXpertQA also provides a rich setting for assessing reasoning abilities beyond mathematics and code, and the authors developed a dedicated reasoning-oriented subset to facilitate evaluation of o1-like reasoning models specifically.

Benchmark Specifications

How MedXpertQA Is Scored

Both the Text and MM subsets are scored as multiple-choice/open-ended accuracy against expert-verified ground truth, reported separately for each subset (and by specialty/body-system where relevant), since text-only and multimodal reasoning represent distinct capability profiles.

State-of-the-Art Results

Scores vary significantly between the Text and MM subsets and by model release. See the paper and official repository for current standings across 18 evaluated models.

MedXpertQA on Benchgen

No Benchgen results yet — be the first to run MedXpertQA.

MedXpertQA vs Other Benchmarks

MedXpertQA targets specialty-board-level clinical reasoning difficulty across all of medicine, complementing narrower imaging-specific benchmarks like MIMIC-CXR (chest radiographs) and PMC-VQA (literature-derived figures).

Run MedXpertQA on Your Model

Benchgen lets teams run both the Text and MM subsets of MedXpertQA against their own model versions on demand, tracking accuracy by specialty and body system to catch regressions in specific areas of clinical reasoning.

Related Models & Benchmarks

Explore related multimodal benchmarks like PMC-VQA and MIMIC-CXR to compare specialty-board-level medical reasoning with imaging-focused medical VQA evaluation.

Frequently Asked Questions