Benchgen

TheoremQA — Results

RankModelScore
1ling-3-0-flash-base61.1
2ling-3-0-tiny-base51.64
T

TheoremQA

1 phaseActive

Theorem-driven QA benchmark testing whether models can apply 350+ theorems (math, physics, EE/CS, finance) to solve challenging science problems.

Overview

TheoremQA

Category Metric Saturation Created

Paper GitHub

Quick answer: TheoremQA (Chen et al., 2023, EMNLP) is a theorem-driven question-answering dataset of 800 expert-curated questions requiring the application of 350+ theorems across Math, Physics, EE/CS, and Finance. InclusionAI's Ling-3.0-flash-base scores 61.10% accuracy as of 2026.

At a Glance

What it tests: Whether a model can identify and correctly apply the right theorem (e.g. Taylor's theorem, Lagrange's theorem, Huffman coding, Quantum Theorem, Elasticity Theorem) to solve a challenging, domain-specific science or finance problem.

Why it matters: Grade-school-style math benchmarks like GSM8K are largely saturated by frontier models, but applying specific, named theorems to novel problems requires deeper domain knowledge — TheoremQA's original paper found GPT-4 topped out at 51% accuracy with Program-of-Thoughts prompting, and all open-source models of the time stayed below 15%, barely above random guessing.

Known limitations: The dataset spans only 800 questions across a broad span of domains, so per-domain sample sizes are small; scores cited here are self-reported by model developers rather than pulled from a shared third-party leaderboard.

What TheoremQA Measures

TheoremQA was introduced as the first theorem-driven question-answering dataset, curated by domain experts to evaluate whether AI models can apply theorems — not just recall facts or do arithmetic — to solve challenging problems spanning Math, Physics, Electrical Engineering & Computer Science, and Finance.

Benchmark Specifications

FieldValue
Task categoryReasoning (theorem application)
Metric% accuracy
Number of questions800
Theorems covered350+
DomainsMath, Physics, EE/CS, Finance
Created byChen et al.
OrgUniversity of Waterloo
Source paperChen et al. 2023 (EMNLP)
GitHubTIGER-AI-Lab/TheoremQA

How TheoremQA Is Scored

Models are scored on accuracy — the percentage of the 800 questions answered correctly, typically evaluated with Chain-of-Thought or Program-of-Thought prompting strategies.

State-of-the-Art Results

Scores above are self-reported by InclusionAI on their respective model cards and shown for context. They are not Benchgen measurements, and are not drawn from a shared third-party leaderboard — only the 2 scores above are currently tracked locally.

TheoremQA on Benchgen

No Benchgen results yet — be the first to run TheoremQA.

TheoremQA vs Other Benchmarks

BenchmarkWhat it testsSaturation
TheoremQATheorem-application reasoning across math/physics/EE-CS/financeLow
MMLU-ProBroad multi-domain knowledge and reasoningMedium
GSM8KGrade-school arithmetic word problemsHigh

TheoremQA is distinct from arithmetic-style math benchmarks in that it requires identifying and applying a specific named theorem, giving a sharper read on domain-specific reasoning depth rather than general math fluency.

Run TheoremQA on Your Model

Benchgen lets you evaluate your own model against TheoremQA's theorem-application problem set and track results across model versions.

Frequently Asked Questions

What is TheoremQA? TheoremQA is a theorem-driven question-answering dataset of 800 expert-curated questions requiring the application of 350+ theorems across Math, Physics, EE/CS, and Finance — introduced by Wenhu Chen and collaborators (University of Waterloo), accepted at EMNLP 2023.
What does a good TheoremQA score look like? InclusionAI's Ling-3.0-flash-base reports 61.10% accuracy as of 2026, the highest currently tracked on Benchgen; Ling-3.0-tiny-base, a smaller sibling model, scores 51.64%. For context, the original paper found GPT-4 reached only 51% with Program-of-Thoughts prompting, and contemporary open-source models stayed below 15%.
Who created TheoremQA? TheoremQA was created by Wenhu Chen and collaborators at the University of Waterloo; see the original paper.
Is TheoremQA saturated? No — TheoremQA remains a low-saturation benchmark, with scores reported by individual labs rather than aggregated broadly on a public third-party leaderboard yet.

Benchmark definition paraphrased from Chen et al. 2023. State-of-the-art scores sourced from published model cards and attributed inline. Last updated 2026-11-03.