BigCodeBench-Hard — Results
| Rank | Model | Score |
|---|---|---|
| 1 | qwen2-5-coder-32b-instruct | 0.27 |
B
BigCodeBench-Hard
1 phaseActive
Challenging subset of 148 BigCodeBench tasks requiring complex multi-library function calls across 7 domains.
Overview
BigCodeBench-Hard
Quick answer: BigCodeBench-Hard (Zhuo et al., 2024) is a subset of 148 challenging programming tasks requiring diverse function calls from multiple libraries, testing compositional reasoning on complex software engineering problems.
Benchmark Details
| Property | Value |
|---|---|
| Tasks | 148 challenging programming tasks |
| Libraries | 139 Python libraries (avg 2.8 per task) |
| Domains | 7 (networking, data analysis, visualization, etc.) |
| Metric | Pass rate (test execution) |
Source: Zhuo et al. 2024. Last updated 2026-07-24.