GPQA Diamond

GPQA Diamond

Quick answer: GPQA Diamond is a 198-question multiple-choice benchmark of PhD-level biology, chemistry, and physics questions, created by Rein et al. (2023). Questions are validated by domain experts and designed so that non-expert humans score below 40%; PhD-level domain experts score around 65–74%. As of June 2026, both Fugu and Fugu Ultra reach 95.5%, matching the frontier.

At a Glance

What it tests: Graduate-level scientific knowledge and reasoning across biology, chemistry, and physics — questions that require PhD-level domain expertise to answer correctly.

Why it matters: GPQA Diamond has become the industry standard for measuring graduate-level scientific reasoning. Every major lab reports it; passing it well (>85%) indicates genuine expert-level understanding, not pattern matching.

Known limitations: At 198 questions, GPQA Diamond has high score variance — a few questions answered differently can shift a model's score by 1–2 percentage points. Top models now cluster above 90%, and it is beginning to approach saturation at the frontier.

What GPQA Diamond Measures

GPQA (Graduate-Level Google-Proof Q&A) was designed to be difficult even for well-informed non-experts with internet access. The "Diamond" subset is the hardest tier — 198 questions where expert validators rated the question as having a clear, unambiguous correct answer that non-experts consistently fail on. Each question is four-choice multiple choice.

To ensure quality, each GPQA Diamond question was written by a PhD student or researcher and then validated by multiple domain experts who agreed it was (a) correct, (b) unambiguous, and (c) genuinely hard. Non-expert humans with PhDs in adjacent fields score around 34%; domain experts score 65–74%. This calibration makes GPQA Diamond one of the most informative hard-science benchmarks available.

By 2026, frontier models approach 95%+, suggesting the top of the leaderboard is beginning to saturate on this benchmark.

Benchmark Specifications

How GPQA Diamond Is Scored

Each question is 4-choice multiple choice; a model scores 1 point per correct answer. The score is reported as a percentage. Due to the small 198-question set, standard error is relatively high (~3.5pp at 95% CI for a 50% score). Scores above 85% are considered strong; scores above 90% are frontier-class. Many labs use chain-of-thought prompting; some use majority voting or extended thinking, which can add a few percentage points.

State-of-the-Art Results

Scores attributed to respective model providers. Sakana Fugu scores use the evaluation conditions in the Fugu technical report.