BIG-Bench Hard — Results
| Rank | Model | Score |
|---|---|---|
| 1 | claude-3-5-sonnet | 0.931 |
| 2 | gemini-1-5-pro | 0.892 |
| 3 | claude-3-opus | 0.868 |
| 4 | gemini-1-5-flash | 0.855 |
| 5 | phi-3-5-moe-instruct | 0.791 |
| 6 | phi-4-mini | 0.704 |
| 7 | phi-3-5-mini-instruct | 0.69 |
BIG-Bench Hard
1 phaseActive
23 challenging BIG-Bench tasks where prior models failed to exceed human performance. 6,511 examples testing multi-step reasoning, logic, and language understanding. Metric: accuracy.
Overview
BIG-Bench Hard
Quick answer: BIG-Bench Hard (BBH) is a curated subset of 23 tasks from the BIG-Bench benchmark suite, selected because they were the tasks where large language models previously failed to exceed average human-rater performance. Created by Suzgun et al. (2022), BBH contains 6,511 examples testing multi-step reasoning, logical deduction, arithmetic, temporal reasoning, and language understanding. It became a key benchmark for evaluating chain-of-thought (CoT) prompting effectiveness.
At a Glance
What it tests: Multi-step reasoning across 23 diverse task types — Boolean logic, logical deduction, geometric reasoning, object counting, causal judgment, temporal sequences, navigation, and more. Tasks were chosen specifically because standard prompting failed; CoT prompting often significantly improves performance.
Why it matters: BBH was designed to be the hardest tasks in BIG-Bench — tasks that frontier models in 2022 couldn't solve without explicit chain-of-thought reasoning. It has been widely used to evaluate the impact of instruction tuning, RLHF, and reasoning improvements.
Known limitations: As of 2024–2025, frontier models score 85–93% on BBH, indicating significant progress and partial saturation. The 23 tasks don't cover all reasoning domains and may underrepresent multimodal or coding reasoning.
What BIG-Bench Hard Measures
BBH includes 23 tasks covering diverse reasoning types:
- Logical reasoning: Boolean expressions, logical deduction, dyck languages
- Mathematical reasoning: Multistep arithmetic, word sorting
- Spatial reasoning: Geometric shapes, spatial navigation
- Temporal reasoning: Date understanding, temporal sequences
- Language understanding: Causal judgment, disambiguation QA, snarky descriptions
- Object tracking: Object counting, tracking shuffled objects
Each task is presented as a multiple-choice or free-response question. The benchmark is typically evaluated with chain-of-thought (CoT) prompting, where models are prompted to reason step-by-step before giving a final answer.
Benchmark Specifications
| Field | Value |
|---|---|
| Tasks | 6,511 examples |
| Subtasks | 23 |
| Primary metric | Accuracy (% correct) |
| Evaluation method | CoT prompting |
| Created by | Suzgun et al. |
| Paper | arXiv:2210.09261 (Oct 2022) |
| Saturation | Medium — frontier models 85–93% |
Frequently Asked Questions
Why does BIG-Bench Hard focus on CoT prompting? BBH tasks were specifically selected because they required multi-step reasoning that standard few-shot prompting couldn't reliably solve. Chain-of-thought prompting showed dramatic improvements on these tasks, making BBH a key benchmark for evaluating CoT techniques.
What score do frontier models get on BIG-Bench Hard? Claude 3.5 Sonnet leads at 93.1%. Gemini 1.5 Pro scores 89.2%. Open-weight models like Gemma 3 27B reach 87.6%. Smaller models typically score 60–80% with CoT.
Is BIG-Bench Hard still used? BBH remains a standard evaluation benchmark, though newer benchmarks like GPQA, HLE, and ARC-AGI have taken over as the primary frontier-model difficulty measures. BBH is particularly useful for evaluating reasoning improvements at the mid-tier model scale.