ProfBench

ProfBench

Quick answer: ProfBench (Wang, Jung, Lu, Diao, Evans, Zeng, Molchanov, Choi, Kautz, Dong et al., NVIDIA, ICLR 2026) evaluates LLMs on 7,000+ response-criterion rubric pairs graded by human domain experts across four professional domains — Physics PhD, Chemistry PhD, Finance MBA, and Consulting MBA. IBM's Granite 4.2 30B scores 42.90% as of October 2026.

At a Glance

What it tests: Whether a model's free-form responses to professional-grade prompts satisfy expert-written rubric criteria across Physics PhD, Chemistry PhD, Finance MBA, and Consulting MBA domains — a mix of generation quality and judgment, not multiple-choice recall.

Why it matters: Most knowledge benchmarks (MMLU-Pro, GPQA) use multiple-choice questions; ProfBench instead grades open-ended, professional-grade outputs against expert rubrics, closer to how a domain specialist would actually evaluate a colleague's work product.

Known limitations: Rubric-based grading of open-ended responses is more expensive to run and score than multiple-choice formats, and IBM's reported scores are self-reported single overall percentages with no public per-domain breakdown — so cross-lab comparisons on this page track only the aggregate metric for now.

What ProfBench Measures

ProfBench was built to close a gap in LLM knowledge evaluation: existing professional-knowledge benchmarks tend to rely on multiple-choice questions that can be gamed or guessed, rather than testing whether a model can produce a response an expert would actually accept. ProfBench instead pairs 7,000+ professional-grade prompts with expert-authored response-criterion rubrics spanning four high-stakes domains — Physics PhD, Chemistry PhD, Finance MBA, and Consulting MBA — and scores model outputs against those rubrics using human expert judges.

Because it requires expert human grading rather than automated string-matching, ProfBench is more expensive to run at scale than most LLM benchmarks, keeping it comparatively low-saturation and under-reported outside of major model-card disclosures.

Benchmark Specifications

How ProfBench Is Scored

Each of the 7,000+ response-criterion pairs is graded by a human domain expert against a rubric item (did the model's response satisfy this specific criterion?); the overall score is the percentage of rubric criteria satisfied across the four domains (Physics PhD, Chemistry PhD, Finance MBA, Consulting MBA).

State-of-the-Art Results

Scores above are self-reported by IBM Research on the official Hugging Face launch blog and shown for context. They are not Benchgen measurements, and are not drawn from a shared third-party leaderboard — only the 3 scores above are currently tracked locally.

ProfBench on Benchgen

No Benchgen results yet — be the first to run ProfBench.

ProfBench vs Other Benchmarks

ProfBench is narrower in domain spread but deeper in grading rigor than multiple-choice knowledge benchmarks like MMLU-Pro or GPQA Diamond — use it when you need to assess whether a model's free-form professional output would satisfy an actual domain expert, not just whether it can pick the right multiple-choice answer.

Run ProfBench on Your Model

Benchgen lets you evaluate your own model against ProfBench's 7,000+ expert-graded professional-knowledge rubric items and track results across model versions.

Related Models & Benchmarks

• Top performers: Granite 4.2 30B · Granite 4.2 8B · Granite 4.2 3B
• Related benchmarks: MMLU-Pro · GPQA Diamond

Frequently Asked Questions

Benchmark definition paraphrased from Wang et al. 2026. State-of-the-art scores sourced from published model cards and attributed inline. Last updated 2026-10-31.