CodeForces

CodeForces (LLM Benchmark)

Quick answer: "Codeforces as an LLM benchmark" means judging models on real competitive-programming problems from Codeforces under contest-like rules, reported as a human-comparable Elo rating. There is no single official cross-lab leaderboard — the cleanest standardized public methodology is CodeElo (Quan et al., 2025, Qwen Team), where OpenAI o1 (1217) leads at 2061 Elo (96.6th percentile among human competitors).

At a Glance

What it tests: Whether a model can solve genuine, difficulty-rated competitive-programming problems under rules and scoring comparable to human contestants. Why it matters: Codeforces problems have hidden tests and contest-level difficulty far beyond typical code-generation benchmarks, making Elo one of the clearest public signals of algorithmic coding skill. Known limitations: Different labs report Codeforces-based scores using different protocols (contest simulation windows, submission limits, problem sets, and Elo calculation methods) — scores from different sources are not directly comparable. See the caveat below.

What CodeForces (as an LLM Benchmark) Measures

Rather than a single official benchmark, "Codeforces" as an LLM eval refers to a family of related methodologies that all use real Codeforces contest problems and score models with a human-comparable Elo rating (the same rating system Codeforces uses for competitors). The most rigorously documented public version is CodeElo (Quan et al., 2025), which draws 387 problems from 54 contests held between May and November 2024, submits solutions directly to the Codeforces judge, and computes Elo using the platform's own rating formula.

Other widely cited numbers — including OpenAI's o1 announcement — come from internal simulations of Codeforces-style contests rather than the CodeElo protocol, and use different problem sets and submission rules.

⚠️ Protocol Caveat

Codeforces-based LLM scores are not standardized across sources. Before comparing two numbers, check the protocol column below:

• CodeElo protocol (Quan et al., 2025): 387 problems, 54 contests (May–Nov 2024), direct submission to the Codeforces judge, official Elo formula.
• OpenAI internal simulation: contest-style problems evaluated internally by OpenAI, reported in the o1 announcement — not independently reproducible with the same problem set.
• DeepSeek-R1 comparison table: figures reported in DeepSeek's own release materials, sourced from a mix of the above.

Mixing rows from different protocols into one ranking will overstate or understate real differences — Benchgen presents them side by side with the protocol labeled rather than merging into a single score.

Benchmark Specifications

How CodeForces LLM Evals Are Scored

Models submit solutions to real contest problems, which are judged against the same hidden test cases used for human competitors. Performance across problems is converted into an Elo rating using (in CodeElo's case) Codeforces' own rating formula, plus the percentile that rating corresponds to among the platform's human user base.

State-of-the-Art Results

Protocol column shows which methodology produced each score — see the caveat above before comparing across rows.

CodeForces on Benchgen

No Benchgen results yet — be the first to run a Codeforces-style eval.

CodeForces vs Other Benchmarks

Run CodeForces-Style Evals on Your Model

Benchgen supports version-controlled, regression-tracked competitive-programming evaluations, so you can confirm a model's coding gains hold up on genuinely novel contest problems rather than a single reported Elo figure.

Related Models & Benchmarks

• DeepSeek-R1
• LiveCodeBench
• BigCodeBench

Frequently Asked Questions