# GPQA Chemistry ![Category](https://img.shields.io/badge/Category-Reasoning-1f6feb) ![Metric](https://img.shields.io/badge/Metric-Accuracy-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-214-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-medium-d29922) ![Created](https://img.shields.io/badge/Created-Nov%202023-555) [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/abs/2311.12022) [![GitHub](https://img.shields.io/badge/GitHub-idavidrein%2Fgpqa-181717?logo=github&logoColor=white)](https://github.com/idavidrein/gpqa) > **Quick answer:** GPQA Chemistry is the chemistry-subdomain slice of GPQA, a dataset of graduate-level, "Google-proof" multiple-choice questions written and validated by PhD-level domain experts, isolating organic, inorganic, analytical, and physical chemistry questions so a model's chemistry reasoning can be assessed independently of its physics or biology performance. ## At a Glance **What it tests:** Whether a model can answer PhD-level chemistry questions that are difficult even for highly skilled non-expert chemists (and physicists/biologists) with unrestricted internet access. **Why it matters:** Aggregate GPQA scores can mask large per-domain gaps; isolating chemistry lets researchers see whether a model's overall GPQA score is being propped up by strength in physics or biology while chemistry reasoning lags. **Known limitations:** At 214 questions (from GPQA's 546-question extended set), the chemistry subset is small, and — like the rest of GPQA — the dataset is intentionally kept from public disclosure in plain text to reduce training-data leakage, which limits independent auditing. ## What GPQA Chemistry Measures GPQA's full dataset spans biology, physics, and chemistry, with each question written by a domain expert and validated by other PhD-level experts to confirm it is both objective and "Google-proof" — meaning skilled non-experts with full internet access still cannot reliably answer it. The paper's domain breakdown (Table 3 of Rein et al. 2023) reports 214 chemistry questions in the 546-question extended set, split across subdomains including Organic Chemistry (144), Chemistry (general) (64), Inorganic Chemistry (3), Analytical Chemistry (2), and Physical Chemistry (1). Because GPQA's chemistry questions skew heavily toward organic chemistry, this subset is especially useful for teams that want to probe a model's synthesis, reaction-mechanism, and structural-reasoning ability specifically, rather than its aggregate science knowledge. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Reasoning / graduate-level chemistry Q&A | | Metric | Accuracy (%) on 4-choice multiple-choice questions | | Number of tasks | 214 chemistry questions (extended-set breakdown) | | Saturation | Medium — GPT-4 few-shot CoT scored 31.8% on chemistry questions in the original paper, versus PhD-level expert accuracy of 65%+ | | Created by | David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, Samuel R. Bowman, et al. (NYU, Anthropic, Cohere) | | Source paper | [Rein et al. 2023](https://arxiv.org/abs/2311.12022) | | GitHub | [idavidrein/gpqa](https://github.com/idavidrein/gpqa) | ## How GPQA Chemistry Is Scored Each question is a 4-choice multiple-choice item; accuracy is the percentage of chemistry questions answered correctly. Random guessing yields 25% accuracy, PhD-level expert chemists average roughly 65% (74% when discounting acknowledged mistakes), and skilled non-expert validators with unrestricted internet access average only around 34%. ## State-of-the-Art Results > In the original paper, GPT-4 with few-shot chain-of-thought prompting scored 31.8% accuracy on chemistry questions specifically — notably lower than its accuracy on biology questions (58.1%), underscoring that chemistry is one of GPQA's hardest subdomains for LLMs. Newer frontier reasoning models have since pushed GPQA Diamond accuracy well above 90%, but chemistry-specific breakdowns for the newest models are less commonly reported. ## GPQA Chemistry on Benchgen > No Benchgen results yet — [be the first to run GPQA Chemistry](/benchmarks/david-rein-betty-li-hou-asa-cooper-stickland-jackson-petty-richard-yuanzhe-pang-julien-dirani-julian-michael-samuel-r-bowman-et-al-nyu-anthropic-cohere/gpqa-chemistry). ## GPQA Chemistry vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **GPQA Chemistry** | PhD-level chemistry reasoning, Google-proof | 214 | medium | | [GPQA Biology](/benchmarks/david-rein-betty-li-hou-asa-cooper-stickland-jackson-petty-richard-yuanzhe-pang-julien-dirani-julian-michael-samuel-r-bowman-et-al-nyu-anthropic-cohere/gpqa-biology) | PhD-level biology reasoning, Google-proof | 105 | medium | | [GPQA Diamond](/benchmarks/rein-et-al/gpqa-diamond) | Highest-quality cross-domain GPQA subset | 198 | high | GPQA Chemistry differs from GPQA Diamond by isolating a single scientific domain rather than mixing physics, biology, and chemistry, making it better suited for diagnosing domain-specific weaknesses rather than reporting one aggregate science-reasoning score. ## Run GPQA Chemistry on Your Model Benchgen lets teams evaluate their own model against the chemistry-specific slice of GPQA, surfacing whether strong aggregate GPQA performance is being driven by physics/biology strength while chemistry reasoning lags behind. ## Frequently Asked Questions

What is GPQA Chemistry?

GPQA Chemistry is the chemistry-subdomain subset of GPQA, a graduate-level, Google-proof multiple-choice benchmark written and validated by PhD-level chemistry experts.

What does a good GPQA Chemistry score look like?

In the original 2023 paper, GPT-4 with few-shot chain-of-thought scored 31.8% on chemistry questions, well below the ~65%+ accuracy of PhD-level expert chemists, so scores meaningfully above the low-30s percent range represent genuine progress.

Who created GPQA Chemistry?

GPQA (including its chemistry subdomain) was created by David Rein and collaborators at NYU, Anthropic, and Cohere, published in November 2023.

GPQA Chemistry — Phases

GPQA Chemistry