# CodeForces (LLM Benchmark) ![Category](https://img.shields.io/badge/Category-Coding-1f6feb) ![Metric](https://img.shields.io/badge/Metric-Elo%20Rating-2da44e) ![Problems](https://img.shields.io/badge/Problems-387-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-Low-red) ![Created](https://img.shields.io/badge/Created-Jan%202025-555) [![Paper](https://img.shields.io/badge/Paper-arXiv%202501.01257-b31b1b)](https://arxiv.org/abs/2501.01257) > **Quick answer:** "Codeforces as an LLM benchmark" means judging models on real competitive-programming problems from Codeforces under contest-like rules, reported as a human-comparable Elo rating. There is no single official cross-lab leaderboard — the cleanest standardized public methodology is CodeElo (Quan et al., 2025, Qwen Team), where OpenAI o1 (1217) leads at 2061 Elo (96.6th percentile among human competitors). --- ## At a Glance **What it tests:** Whether a model can solve genuine, difficulty-rated competitive-programming problems under rules and scoring comparable to human contestants. **Why it matters:** Codeforces problems have hidden tests and contest-level difficulty far beyond typical code-generation benchmarks, making Elo one of the clearest public signals of algorithmic coding skill. **Known limitations:** Different labs report Codeforces-based scores using different protocols (contest simulation windows, submission limits, problem sets, and Elo calculation methods) — scores from different sources are **not directly comparable**. See the caveat below. ## What CodeForces (as an LLM Benchmark) Measures Rather than a single official benchmark, "Codeforces" as an LLM eval refers to a family of related methodologies that all use real Codeforces contest problems and score models with a human-comparable Elo rating (the same rating system Codeforces uses for competitors). The most rigorously documented public version is CodeElo (Quan et al., 2025), which draws 387 problems from 54 contests held between May and November 2024, submits solutions directly to the Codeforces judge, and computes Elo using the platform's own rating formula. Other widely cited numbers — including OpenAI's o1 announcement — come from internal simulations of Codeforces-style contests rather than the CodeElo protocol, and use different problem sets and submission rules. ## ⚠️ Protocol Caveat Codeforces-based LLM scores are **not standardized across sources**. Before comparing two numbers, check the protocol column below: - **CodeElo protocol** (Quan et al., 2025): 387 problems, 54 contests (May–Nov 2024), direct submission to the Codeforces judge, official Elo formula. - **OpenAI internal simulation**: contest-style problems evaluated internally by OpenAI, reported in the [o1 announcement](https://openai.com/index/learning-to-reason-with-llms/) — not independently reproducible with the same problem set. - **DeepSeek-R1 comparison table**: figures reported in DeepSeek's own release materials, sourced from a mix of the above. Mixing rows from different protocols into one ranking will overstate or understate real differences — Benchgen presents them side by side with the protocol labeled rather than merging into a single score. ## Benchmark Specifications | Field | Value | |---|---| | Problems (CodeElo protocol) | 387, from 54 contests | | Contest window (CodeElo) | May 4, 2024 – Nov 4, 2024 | | Metric | Elo rating + percentile among human competitors | | Task category | Competitive programming / coding | | Released | January 2025 (CodeElo); OpenAI internal reporting from September 2024 | | Created by | Quan, Yang, Yu, Zheng, Liu, Yang, Ren, Gao, Miao, Feng, Wang, Yang, Cui, Fan, Zhang, Hui, Lin (Qwen Team, Alibaba Group) | ## How CodeForces LLM Evals Are Scored Models submit solutions to real contest problems, which are judged against the same hidden test cases used for human competitors. Performance across problems is converted into an Elo rating using (in CodeElo's case) Codeforces' own rating formula, plus the percentile that rating corresponds to among the platform's human user base. ## State-of-the-Art Results | Model | Score | Protocol | Source | |---|---:|---|---| | OpenAI o1 (1217) | 2061 Elo (96.6 pct) | DeepSeek-R1 comparison table | [DeepSeek-R1 repo](https://github.com/deepseek-ai/DeepSeek-R1) | | DeepSeek-R1 | 2029 Elo (96.3 pct) | DeepSeek-R1 comparison table | [DeepSeek-R1 repo](https://github.com/deepseek-ai/DeepSeek-R1) | | o1-mini | 1820 Elo (93.4 pct) | DeepSeek-R1 comparison table | [DeepSeek-R1 repo](https://github.com/deepseek-ai/DeepSeek-R1) | | OpenAI o1-ioi | 1807 Elo (93 pct) | OpenAI internal simulation | [OpenAI o1 post](https://openai.com/index/learning-to-reason-with-llms/) | | OpenAI o1 | 1673 Elo (89 pct) | OpenAI internal simulation | [OpenAI o1 post](https://openai.com/index/learning-to-reason-with-llms/) | | o1-mini | 1578 Elo (89.2 pct) | CodeElo | [CodeElo paper](https://arxiv.org/html/2501.01257v2) | | QwQ-32B-Preview | 1261 Elo (63.6 pct) | CodeElo | [CodeElo paper](https://arxiv.org/html/2501.01257v2) | | OpenAI o1-preview | 1258 Elo (62 pct) | OpenAI internal simulation | [OpenAI o1 post](https://openai.com/index/learning-to-reason-with-llms/) | | DeepSeek-V3 | 1134 Elo (58.7 pct) | DeepSeek-R1 comparison table | [DeepSeek-R1 repo](https://github.com/deepseek-ai/DeepSeek-R1) | | Claude 3.5 Sonnet (2024-10-22) | 710 Elo (24.1 pct) | CodeElo | [CodeElo paper](https://arxiv.org/html/2501.01257v2) | *Protocol column shows which methodology produced each score — see the caveat above before comparing across rows.* ## CodeForces on Benchgen > No Benchgen results yet — [be the first to run a Codeforces-style eval](/benchmarks/codeforces). ## CodeForces vs Other Benchmarks | Benchmark | What it tests | Saturation | |---|---|---| | CodeForces (LLM benchmark) | Competitive programming, human-comparable Elo | Low | | [LiveCodeBench](/benchmarks/livecodebench) | Contamination-resistant coding benchmark | — | | [BigCodeBench](/benchmarks/bigcodebench) | Practical, library-heavy code generation | — | ## Run CodeForces-Style Evals on Your Model Benchgen supports version-controlled, regression-tracked competitive-programming evaluations, so you can confirm a model's coding gains hold up on genuinely novel contest problems rather than a single reported Elo figure. ## Related Models & Benchmarks - [DeepSeek-R1](/models/deepseek/deepseek-r1-0528) - [LiveCodeBench](/benchmarks/livecodebench) - [BigCodeBench](/benchmarks/bigcodebench) ## Frequently Asked Questions

What is the "Codeforces" LLM benchmark?

It's shorthand for evaluating models on real Codeforces competitive-programming problems and reporting a human-comparable Elo rating. There's no single official cross-lab benchmark — CodeElo (Quan et al., 2025) is the most standardized public protocol.

Why do different sources report different Codeforces scores for the same model?

Because protocols differ: problem sets, contest simulation rules, submission limits, and Elo calculation methods vary by source. Always check the protocol column before comparing two numbers.

Who created CodeElo?

CodeElo was created by Shanghaoran Quan and collaborators at the Qwen Team, Alibaba Group, published in January 2025.

CodeForces — Results

CodeForces — Evaluations