AetherCode

AetherCode

Quick answer: AetherCode is ByteDance's benchmark of olympiad-level competitive programming problems from IOI/ICPC contests, testing LLMs on advanced algorithmic reasoning and code synthesis. Best Pass@1: 35.5% (o4-mini-high, 2025). Even top models solve fewer than 40% of problems.

At a Glance

What it tests: An LLM's ability to solve hard algorithmic coding problems drawn from competitive programming contests — requiring mastery of data structures, graph algorithms, dynamic programming, number theory, and combinatorics.

Why it matters: Standard coding benchmarks (HumanEval, MBPP, LiveCodeBench) have saturated for frontier models. AetherCode's olympiad-difficulty floor creates meaningful separation between models that can reason through complex multi-step algorithmic problems and those that cannot, making it relevant for evaluating state-of-the-art reasoning-heavy code generation.

Known limitations: No paper or public dataset is available as of mid-2026, limiting independent reproduction and analysis. With only 2 models tracked publicly, leaderboard coverage is sparse.

What AetherCode Measures

AetherCode focuses on the highest tier of coding difficulty: problems from programming olympiads, where solutions require not just syntactically correct code but deep algorithmic insight. Unlike benchmarks built from interview-style questions or library usage tasks, olympiad problems demand that a model understand and apply non-trivial algorithms correctly under problem-specific constraints.

The benchmark evaluates whether a model can derive the correct algorithmic approach, implement it efficiently, and handle edge cases — all in a single pass without interactive debugging. This makes it a direct measure of end-to-end algorithmic problem-solving rather than code autocomplete or library API recall.

Scores are reported on a 0–1 scale, reflecting the fraction of problems solved correctly.

Benchmark Specifications

How AetherCode Is Scored

Models are evaluated by executing their generated code against hidden test cases for each problem. A problem is marked correct only if the solution passes all test cases. The aggregate score is the fraction of problems solved, reported on a 0–1 scale where 1.0 means all olympiad problems solved correctly.

AetherCode on Benchgen

No Benchgen results yet — be the first to run AetherCode.

AetherCode vs Other Benchmarks

AetherCode sits at the hardest end of the coding difficulty spectrum alongside LiveCodeBench Pro. Use AetherCode when you need to differentiate frontier models on pure algorithmic reasoning; use SWE-Bench Pro when production engineering tasks matter more.

Run AetherCode on Your Model

Benchgen lets you run AetherCode against your own model checkpoints and track performance across fine-tuning runs, catching regressions in algorithmic reasoning before they reach production. Version-controlled benchmark results replace one-off comparisons with a continuous quality signal.

Frequently Asked Questions