# OJBench ![Category](https://img.shields.io/badge/Category-Coding-1f6feb) ![Metric](https://img.shields.io/badge/Metric-Pass%20Rate-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-232-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-low-d29922) ![Created](https://img.shields.io/badge/Created-Jun%202025-555) [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/abs/2506.16395) [![GitHub](https://img.shields.io/badge/GitHub-He--Ren%2FOJBench-181717?logo=github&logoColor=white)](https://github.com/He-Ren/OJBench) > **Quick answer:** OJBench is a competition-level code reasoning benchmark of 232 rigorously-selected problems drawn from China's National Olympiad in Informatics (NOI) and the International Collegiate Programming Contest (ICPC), designed to test LLMs against genuinely elite human programming-competition difficulty rather than typical interview-style coding problems. ## At a Glance **What it tests:** Whether a model can solve real, judge-verified competitive-programming problems at the difficulty level of top human programming olympiads, going well beyond the LeetCode-style difficulty of most existing code benchmarks. **Why it matters:** Existing code benchmarks rarely probe competition-level difficulty, so strong performance elsewhere can mask a model's actual ceiling on the hardest algorithmic problems that separate expert human programmers. **Known limitations:** At only 232 problems, OJBench is far smaller than benchmarks like HumanEval+ or MBPP+, and its focus on elite competitive programming means it measures a narrower (though harder) slice of coding ability than general software-engineering benchmarks. ## What OJBench Measures OJBench sources its 232 problems exclusively from human programming contests — China's National Olympiad in Informatics (NOI) and the International Collegiate Programming Contest (ICPC) — rather than generating or adapting problems synthetically. This grounds the benchmark's difficulty in problems that were originally designed to challenge the strongest competitive human programmers, giving OJBench a meaningfully higher difficulty ceiling than typical LLM code benchmarks. The authors evaluated 37 models spanning both closed-source and open-source systems, and both reasoning-oriented and non-reasoning-oriented architectures. Their central finding is sobering: even state-of-the-art reasoning-focused models such as o4-mini and Gemini-2.5-Pro-exp struggle significantly with the hardest problems in the set, underscoring a persistent gap between current LLM capabilities and genuine competitive-level code reasoning. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Coding / competition-level code reasoning | | Metric | Pass rate (%) on judge-verified submissions | | Number of tasks | 232 problems from NOI and ICPC | | Saturation | Low — even o4-mini and Gemini-2.5-Pro-exp struggle on the hardest problems | | Created by | Zhexu Wang, Yiping Liu, Yejie Wang, Wenyang He, Bofei Gao, Muxi Diao, Yanxu Chen, Kelin Fu, Flood Sung, et al. (Moonshot AI) | | Source paper | [Wang et al. 2025](https://arxiv.org/abs/2506.16395) | | GitHub | [He-Ren/OJBench](https://github.com/He-Ren/OJBench) | ## How OJBench Is Scored Submissions are graded against the same judge infrastructure used in the original NOI/ICPC competitions, giving a binary pass/fail verdict per problem based on correctness and typically also efficiency constraints (time/memory limits), consistent with real competitive-programming judging. The headline metric is the aggregate pass rate (%) across the 232-problem set. ## State-of-the-Art Results > Across 37 evaluated models, even top reasoning-oriented systems like o4-mini and Gemini-2.5-Pro-exp fail on a significant share of the hardest problems. See the [official GitHub repository](https://github.com/He-Ren/OJBench) and paper for the full per-model breakdown. ## OJBench on Benchgen > No Benchgen results yet — [be the first to run OJBench](/benchmarks/zhexu-wang-yiping-liu-yejie-wang-wenyang-he-bofei-gao-muxi-diao-yanxu-chen-kelin-fu-flood-sung-et-al-moonshot-ai/ojbench). ## OJBench vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **OJBench** | Elite competition-level code reasoning (NOI/ICPC) | 232 | low | | [HumanEval+](/benchmarks/jiawei-liu-chunqiu-steven-xia-yuyao-wang-lingming-zhang-university-of-illinois-urbana-champaign/humaneval) | Rigorous single-function Python generation | 164 | high | | [Multi-SWE-Bench](/benchmarks/daoguang-zan-zhirong-huang-wei-liu-et-al-bytedance/multi-swe-bench) | Multilingual GitHub issue resolution | 1,632 | low | OJBench is distinguished from software-engineering benchmarks like Multi-SWE-Bench and general coding benchmarks like HumanEval+ by testing pure algorithmic problem-solving at the difficulty ceiling of elite human programming competitions, rather than real-world repository or single-function tasks. ## Run OJBench on Your Model Benchgen lets teams evaluate their own model against OJBench's competition-level problem set, surfacing exactly where a model's algorithmic reasoning breaks down relative to elite human competitive programmers. ## Frequently Asked Questions

What is OJBench?

OJBench is a benchmark of 232 competition-level programming problems sourced from NOI and ICPC, designed to test LLMs' code reasoning ability at elite human-competition difficulty.

What does a good OJBench score look like?

Even top reasoning-oriented models like o4-mini and Gemini-2.5-Pro-exp struggle on OJBench's hardest problems, so a meaningfully high pass rate across the full 232-problem set represents strong performance.

Who created OJBench?

OJBench was created by Zhexu Wang and collaborators, published in June 2025.

OJBench — Phases

OJBench