GPQA Chemistry

GPQA Chemistry

Quick answer: GPQA Chemistry is the chemistry-subdomain slice of GPQA, a dataset of graduate-level, "Google-proof" multiple-choice questions written and validated by PhD-level domain experts, isolating organic, inorganic, analytical, and physical chemistry questions so a model's chemistry reasoning can be assessed independently of its physics or biology performance.

At a Glance

What it tests: Whether a model can answer PhD-level chemistry questions that are difficult even for highly skilled non-expert chemists (and physicists/biologists) with unrestricted internet access.

Why it matters: Aggregate GPQA scores can mask large per-domain gaps; isolating chemistry lets researchers see whether a model's overall GPQA score is being propped up by strength in physics or biology while chemistry reasoning lags.

Known limitations: At 214 questions (from GPQA's 546-question extended set), the chemistry subset is small, and — like the rest of GPQA — the dataset is intentionally kept from public disclosure in plain text to reduce training-data leakage, which limits independent auditing.

What GPQA Chemistry Measures

GPQA's full dataset spans biology, physics, and chemistry, with each question written by a domain expert and validated by other PhD-level experts to confirm it is both objective and "Google-proof" — meaning skilled non-experts with full internet access still cannot reliably answer it. The paper's domain breakdown (Table 3 of Rein et al. 2023) reports 214 chemistry questions in the 546-question extended set, split across subdomains including Organic Chemistry (144), Chemistry (general) (64), Inorganic Chemistry (3), Analytical Chemistry (2), and Physical Chemistry (1).

Because GPQA's chemistry questions skew heavily toward organic chemistry, this subset is especially useful for teams that want to probe a model's synthesis, reaction-mechanism, and structural-reasoning ability specifically, rather than its aggregate science knowledge.

Benchmark Specifications

How GPQA Chemistry Is Scored

Each question is a 4-choice multiple-choice item; accuracy is the percentage of chemistry questions answered correctly. Random guessing yields 25% accuracy, PhD-level expert chemists average roughly 65% (74% when discounting acknowledged mistakes), and skilled non-expert validators with unrestricted internet access average only around 34%.

State-of-the-Art Results

In the original paper, GPT-4 with few-shot chain-of-thought prompting scored 31.8% accuracy on chemistry questions specifically — notably lower than its accuracy on biology questions (58.1%), underscoring that chemistry is one of GPQA's hardest subdomains for LLMs. Newer frontier reasoning models have since pushed GPQA Diamond accuracy well above 90%, but chemistry-specific breakdowns for the newest models are less commonly reported.

GPQA Chemistry on Benchgen

No Benchgen results yet — be the first to run GPQA Chemistry.

GPQA Chemistry vs Other Benchmarks

GPQA Chemistry differs from GPQA Diamond by isolating a single scientific domain rather than mixing physics, biology, and chemistry, making it better suited for diagnosing domain-specific weaknesses rather than reporting one aggregate science-reasoning score.

Run GPQA Chemistry on Your Model

Benchgen lets teams evaluate their own model against the chemistry-specific slice of GPQA, surfacing whether strong aggregate GPQA performance is being driven by physics/biology strength while chemistry reasoning lags behind.

Frequently Asked Questions