# ProfBench ![Category](https://img.shields.io/badge/Category-Professional%20Knowledge-1f6feb) ![Metric](https://img.shields.io/badge/Metric-%25%20Accuracy-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-7000%2B-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-Low-d29922) ![Created](https://img.shields.io/badge/Created-Oct%202025-555) [![Paper](https://img.shields.io/badge/Paper-arXiv%202510.18941-b31b1b)](https://arxiv.org/abs/2510.18941) [![GitHub](https://img.shields.io/badge/GitHub-NVlabs%2FProfBench-181717?logo=github&logoColor=white)](https://github.com/NVlabs/ProfBench) > **Quick answer:** ProfBench (Wang, Jung, Lu, Diao, Evans, Zeng, Molchanov, Choi, Kautz, Dong et al., NVIDIA, ICLR 2026) evaluates LLMs on 7,000+ response-criterion rubric pairs graded by human domain experts across four professional domains — Physics PhD, Chemistry PhD, Finance MBA, and Consulting MBA. IBM's Granite 4.2 30B scores 42.90% as of October 2026. ## At a Glance **What it tests:** Whether a model's free-form responses to professional-grade prompts satisfy expert-written rubric criteria across Physics PhD, Chemistry PhD, Finance MBA, and Consulting MBA domains — a mix of generation quality and judgment, not multiple-choice recall. **Why it matters:** Most knowledge benchmarks (MMLU-Pro, GPQA) use multiple-choice questions; ProfBench instead grades open-ended, professional-grade outputs against expert rubrics, closer to how a domain specialist would actually evaluate a colleague's work product. **Known limitations:** Rubric-based grading of open-ended responses is more expensive to run and score than multiple-choice formats, and IBM's reported scores are self-reported single overall percentages with no public per-domain breakdown — so cross-lab comparisons on this page track only the aggregate metric for now. ## What ProfBench Measures ProfBench was built to close a gap in LLM knowledge evaluation: existing professional-knowledge benchmarks tend to rely on multiple-choice questions that can be gamed or guessed, rather than testing whether a model can produce a response an expert would actually accept. ProfBench instead pairs 7,000+ professional-grade prompts with expert-authored response-criterion rubrics spanning four high-stakes domains — Physics PhD, Chemistry PhD, Finance MBA, and Consulting MBA — and scores model outputs against those rubrics using human expert judges. Because it requires expert human grading rather than automated string-matching, ProfBench is more expensive to run at scale than most LLM benchmarks, keeping it comparatively low-saturation and under-reported outside of major model-card disclosures. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Professional knowledge (expert rubric grading) | | Metric | % accuracy (rubric criteria satisfied) | | Number of tasks | 7,000+ response-criterion pairs | | Saturation | Low | | Created by | Wang et al. (NVIDIA) | | Source paper | [Wang et al. 2026](https://arxiv.org/abs/2510.18941) (ICLR 2026) | | GitHub | [NVlabs/ProfBench](https://github.com/NVlabs/ProfBench) | | Dataset | [nvidia/ProfBench on Hugging Face](https://huggingface.co/datasets/nvidia/ProfBench) | ## How ProfBench Is Scored Each of the 7,000+ response-criterion pairs is graded by a human domain expert against a rubric item (did the model's response satisfy this specific criterion?); the overall score is the percentage of rubric criteria satisfied across the four domains (Physics PhD, Chemistry PhD, Finance MBA, Consulting MBA). ## State-of-the-Art Results | Rank | Model | Score | Source | Date | |---|---|---|---|---| | 1 | [Granite 4.2 30B](/models/ibm/granite-4-2-30b) | 42.90% | [Granite 4.2 launch blog](https://huggingface.co/blog/ibm-granite/granite-4-2) | 2026-08 | | 2 | [Granite 4.2 8B](/models/ibm/granite-4-2-8b) | 41.20% | [Granite 4.2 launch blog](https://huggingface.co/blog/ibm-granite/granite-4-2) | 2026-08 | | 3 | [Granite 4.2 3B](/models/ibm/granite-4-2-3b) | 32.10% | [Granite 4.2 launch blog](https://huggingface.co/blog/ibm-granite/granite-4-2) | 2026-08 | *Scores above are self-reported by IBM Research on the official Hugging Face launch blog and shown for context. They are not Benchgen measurements, and are not drawn from a shared third-party leaderboard — only the 3 scores above are currently tracked locally.* ## ProfBench on Benchgen > No Benchgen results yet — [be the first to run ProfBench](/benchmarks/wang-et-al/profbench). ## ProfBench vs Other Benchmarks | Benchmark | What it tests | Saturation | |---|---|---| | **ProfBench** | Expert-rubric-graded professional knowledge across Physics/Chemistry PhD, Finance/Consulting MBA | Low | | [MMLU-Pro](/benchmarks/mmlu-pro) | Broad multiple-choice professional/academic knowledge | — | | [GPQA Diamond](/benchmarks/gpqa-diamond) | Graduate-level multiple-choice science questions | — | ProfBench is narrower in domain spread but deeper in grading rigor than multiple-choice knowledge benchmarks like MMLU-Pro or GPQA Diamond — use it when you need to assess whether a model's free-form professional output would satisfy an actual domain expert, not just whether it can pick the right multiple-choice answer. ## Run ProfBench on Your Model Benchgen lets you evaluate your own model against ProfBench's 7,000+ expert-graded professional-knowledge rubric items and track results across model versions. ## Related Models & Benchmarks - Top performers: [Granite 4.2 30B](/models/ibm/granite-4-2-30b) · [Granite 4.2 8B](/models/ibm/granite-4-2-8b) · [Granite 4.2 3B](/models/ibm/granite-4-2-3b) - Related benchmarks: [MMLU-Pro](/benchmarks/mmlu-pro) · [GPQA Diamond](/benchmarks/gpqa-diamond) ## Frequently Asked Questions

What is ProfBench?

ProfBench is an NVIDIA benchmark (ICLR 2026) of 7,000+ response-criterion pairs graded by human domain experts, testing LLM performance on professional-grade tasks across Physics PhD, Chemistry PhD, Finance MBA, and Consulting MBA domains.

What does a good ProfBench score look like?

IBM's Granite 4.2 30B reports 42.90% as of October 2026, the highest currently tracked on Benchgen; Granite 4.2 8B scores 41.20%, and Granite 4.2 3B scores 32.10%.

Who created ProfBench?

ProfBench was created by Wang, Jung, Lu, Diao, Evans, Zeng, Molchanov, Choi, Kautz, Dong and co-authors at NVIDIA; see the [original paper](https://arxiv.org/abs/2510.18941).

Is ProfBench saturated?

No — ProfBench remains a low-saturation benchmark; its expert-grading requirement makes it more costly to run than most benchmarks, keeping reported scores comparatively rare and the overall ceiling (42.90% best reported so far) well below saturation.

How does ProfBench differ from MMLU-Pro?

MMLU-Pro tests broad academic/professional knowledge via multiple-choice questions, while ProfBench grades open-ended, professional-grade responses against expert-written rubrics across four specific domains (Physics PhD, Chemistry PhD, Finance MBA, Consulting MBA).

--- *Benchmark definition paraphrased from [Wang et al. 2026](https://arxiv.org/abs/2510.18941). State-of-the-art scores sourced from published model cards and attributed inline. Last updated 2026-10-31.*

Rank	Model	Score
1	granite-4-2-30b	42.9
2	granite-4-2-8b	41.2
3	granite-4-2-3b	32.1

ProfBench — Results

ProfBench — Leaderboard

ProfBench Leaderboard