# GPQA Biology ![Category](https://img.shields.io/badge/Category-Reasoning-1f6feb) ![Metric](https://img.shields.io/badge/Metric-Accuracy-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-105-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-medium-d29922) ![Created](https://img.shields.io/badge/Created-Nov%202023-555) [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/abs/2311.12022) [![GitHub](https://img.shields.io/badge/GitHub-idavidrein%2Fgpqa-181717?logo=github&logoColor=white)](https://github.com/idavidrein/gpqa) > **Quick answer:** GPQA Biology is the biology-subdomain slice of GPQA, a dataset of graduate-level, "Google-proof" multiple-choice questions written and validated by PhD-level domain experts, isolating molecular biology and genetics questions so a model's biology reasoning can be assessed independently of its physics or chemistry performance. ## At a Glance **What it tests:** Whether a model can answer PhD-level molecular biology and genetics questions that are difficult even for highly skilled non-expert biologists (and physicists/chemists) with unrestricted internet access. **Why it matters:** Aggregate GPQA scores can mask large per-domain gaps; isolating biology lets researchers see whether a model's overall GPQA score is being propped up by strength in physics or chemistry while biology reasoning lags — notably, biology was GPT-4's strongest domain in the original paper. **Known limitations:** At 105 questions (from GPQA's 546-question extended set), the biology subset is GPQA's smallest domain slice, limiting statistical power for fine-grained model comparisons. ## What GPQA Biology Measures GPQA's full dataset spans biology, physics, and chemistry, with each question written by a domain expert and validated by other PhD-level experts to confirm it is both objective and "Google-proof" — meaning skilled non-experts with full internet access still cannot reliably answer it. The paper's domain breakdown (Table 3 of Rein et al. 2023) reports 105 biology questions in the 546-question extended set, split across two subdomains: Molecular Biology (85) and Genetics (20). Because GPQA's biology questions skew heavily toward molecular biology, this subset is especially useful for teams that want to probe a model's understanding of cellular mechanisms, gene regulation, and molecular pathways specifically, rather than its aggregate science knowledge. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Reasoning / graduate-level biology Q&A | | Metric | Accuracy (%) on 4-choice multiple-choice questions | | Number of tasks | 105 biology questions (extended-set breakdown) | | Saturation | Medium — GPT-4 few-shot CoT scored 58.1% on biology questions in the original paper, its strongest of the three domains, versus PhD-level expert accuracy of 65%+ | | Created by | David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, Samuel R. Bowman, et al. (NYU, Anthropic, Cohere) | | Source paper | [Rein et al. 2023](https://arxiv.org/abs/2311.12022) | | GitHub | [idavidrein/gpqa](https://github.com/idavidrein/gpqa) | ## How GPQA Biology Is Scored Each question is a 4-choice multiple-choice item; accuracy is the percentage of biology questions answered correctly. Random guessing yields 25% accuracy, PhD-level expert biologists average roughly 65% (74% when discounting acknowledged mistakes), and skilled non-expert validators with unrestricted internet access average only around 34%. ## State-of-the-Art Results > In the original paper, GPT-4 with few-shot chain-of-thought prompting scored 58.1% accuracy on biology questions — notably its best-performing domain, well above its physics (37.0%) and chemistry (31.8%) scores. Newer frontier reasoning models have since pushed GPQA Diamond accuracy well above 90%, but biology-specific breakdowns for the newest models are less commonly reported. ## GPQA Biology on Benchgen > No Benchgen results yet — [be the first to run GPQA Biology](/benchmarks/david-rein-betty-li-hou-asa-cooper-stickland-jackson-petty-richard-yuanzhe-pang-julien-dirani-julian-michael-samuel-r-bowman-et-al-nyu-anthropic-cohere/gpqa-biology). ## GPQA Biology vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **GPQA Biology** | PhD-level biology reasoning, Google-proof | 105 | medium | | [GPQA Chemistry](/benchmarks/david-rein-betty-li-hou-asa-cooper-stickland-jackson-petty-richard-yuanzhe-pang-julien-dirani-julian-michael-samuel-r-bowman-et-al-nyu-anthropic-cohere/gpqa-chemistry) | PhD-level chemistry reasoning, Google-proof | 214 | medium | | [GPQA Diamond](/benchmarks/rein-et-al/gpqa-diamond) | Highest-quality cross-domain GPQA subset | 198 | high | GPQA Biology differs from GPQA Diamond by isolating a single scientific domain rather than mixing physics, biology, and chemistry, making it better suited for diagnosing domain-specific weaknesses rather than reporting one aggregate science-reasoning score. ## Run GPQA Biology on Your Model Benchgen lets teams evaluate their own model against the biology-specific slice of GPQA, surfacing whether strong aggregate GPQA performance is being driven by physics/chemistry strength while biology reasoning lags behind. ## Frequently Asked Questions

What is GPQA Biology?

GPQA Biology is the biology-subdomain subset of GPQA, a graduate-level, Google-proof multiple-choice benchmark written and validated by PhD-level biology experts.

What does a good GPQA Biology score look like?

In the original 2023 paper, GPT-4 with few-shot chain-of-thought scored 58.1% on biology questions, its strongest domain, but still well below the ~65%+ accuracy of PhD-level expert biologists.

Who created GPQA Biology?

GPQA (including its biology subdomain) was created by David Rein and collaborators at NYU, Anthropic, and Cohere, published in November 2023.

GPQA Biology — Phases

GPQA Biology