OJBench
1 phaseActive
Competition-level code reasoning benchmark using 232 real NOI/ICPC problems, evaluating 37 models and showing frontier reasoning models still struggle.
Overview
OJBench
Quick answer: OJBench is a competition-level code reasoning benchmark of 232 rigorously-selected problems drawn from China's National Olympiad in Informatics (NOI) and the International Collegiate Programming Contest (ICPC), designed to test LLMs against genuinely elite human programming-competition difficulty rather than typical interview-style coding problems.
At a Glance
What it tests: Whether a model can solve real, judge-verified competitive-programming problems at the difficulty level of top human programming olympiads, going well beyond the LeetCode-style difficulty of most existing code benchmarks.
Why it matters: Existing code benchmarks rarely probe competition-level difficulty, so strong performance elsewhere can mask a model's actual ceiling on the hardest algorithmic problems that separate expert human programmers.
Known limitations: At only 232 problems, OJBench is far smaller than benchmarks like HumanEval+ or MBPP+, and its focus on elite competitive programming means it measures a narrower (though harder) slice of coding ability than general software-engineering benchmarks.
What OJBench Measures
OJBench sources its 232 problems exclusively from human programming contests — China's National Olympiad in Informatics (NOI) and the International Collegiate Programming Contest (ICPC) — rather than generating or adapting problems synthetically. This grounds the benchmark's difficulty in problems that were originally designed to challenge the strongest competitive human programmers, giving OJBench a meaningfully higher difficulty ceiling than typical LLM code benchmarks.
The authors evaluated 37 models spanning both closed-source and open-source systems, and both reasoning-oriented and non-reasoning-oriented architectures. Their central finding is sobering: even state-of-the-art reasoning-focused models such as o4-mini and Gemini-2.5-Pro-exp struggle significantly with the hardest problems in the set, underscoring a persistent gap between current LLM capabilities and genuine competitive-level code reasoning.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Coding / competition-level code reasoning |
| Metric | Pass rate (%) on judge-verified submissions |
| Number of tasks | 232 problems from NOI and ICPC |
| Saturation | Low — even o4-mini and Gemini-2.5-Pro-exp struggle on the hardest problems |
| Created by | Zhexu Wang, Yiping Liu, Yejie Wang, Wenyang He, Bofei Gao, Muxi Diao, Yanxu Chen, Kelin Fu, Flood Sung, et al. (Moonshot AI) |
| Source paper | Wang et al. 2025 |
| GitHub | He-Ren/OJBench |
How OJBench Is Scored
Submissions are graded against the same judge infrastructure used in the original NOI/ICPC competitions, giving a binary pass/fail verdict per problem based on correctness and typically also efficiency constraints (time/memory limits), consistent with real competitive-programming judging. The headline metric is the aggregate pass rate (%) across the 232-problem set.
State-of-the-Art Results
Across 37 evaluated models, even top reasoning-oriented systems like o4-mini and Gemini-2.5-Pro-exp fail on a significant share of the hardest problems. See the official GitHub repository and paper for the full per-model breakdown.
OJBench on Benchgen
No Benchgen results yet — be the first to run OJBench.
OJBench vs Other Benchmarks
| Benchmark | What it tests | Tasks | Saturation |
|---|---|---|---|
| OJBench | Elite competition-level code reasoning (NOI/ICPC) | 232 | low |
| HumanEval+ | Rigorous single-function Python generation | 164 | high |
| Multi-SWE-Bench | Multilingual GitHub issue resolution | 1,632 | low |
OJBench is distinguished from software-engineering benchmarks like Multi-SWE-Bench and general coding benchmarks like HumanEval+ by testing pure algorithmic problem-solving at the difficulty ceiling of elite human programming competitions, rather than real-world repository or single-function tasks.
Run OJBench on Your Model
Benchgen lets teams evaluate their own model against OJBench's competition-level problem set, surfacing exactly where a model's algorithmic reasoning breaks down relative to elite human competitive programmers.