ExploitBench

ExploitBench

Quick answer: ExploitBench treats exploitation as a ladder, not a binary crash/no-crash outcome. It decomposes exploiting a real vulnerability into 16 measurable capability flags — from reaching vulnerable code, through triggering a crash, building sandbox primitives and arbitrary read/write, up to full arbitrary code execution — verified by deterministic oracles across 41 real, hardened V8 JavaScript engine bugs.

At a Glance

What it tests: How far an AI agent can climb a graded exploitation capability ladder against real, production-hardened vulnerabilities in V8 (the JS/WASM engine behind Chrome, Edge, and Node.js) — not just whether it can crash the target.

Why it matters: Prior benchmarks collapse exploitation into a single binary success/failure signal, hiding exactly where models struggle. ExploitBench's 16-flag ladder shows that reaching and crashing vulnerable code is comparatively routine for frontier models, while arbitrary code execution against hardened targets remains an emerging, largely unsolved capability.

Known limitations: Narrow domain — 41 tasks, all V8-specific. The authors note a sharp split between publicly deployed frontier models (where full code execution is rare) and an unnamed private/internal model (which reaches it roughly half the time), suggesting public model results understate the frontier's true capability ceiling.

What ExploitBench Measures

Each of ExploitBench's 41 instances is a real V8 bug, chosen specifically because V8 is both widely deployed and heavily exploitation-hardened. Rather than scoring a single "did it work" outcome, the benchmark decomposes exploitation into 16 measurable flags spanning five tiers: coverage, bug triggering, V8-internal primitives, general memory-corruption primitives, and arbitrary code execution. Each flag is verified by a deterministic oracle — using per-run randomized challenge-response checks for primitives, differential execution against ground-truth binaries to measure progress, and a signal-handler proof for code execution — so results can't be gamed by a lucky crash.

The paper reports results across three evaluation arms: <model, environment> as the primary measure of model-environment capability, <model, environment, adaptive coaching> to test whether targeted feedback shifts outcomes, and <model, environment, harness> as an ablation swapping in a vendor's native CLI harness to check for vendor-side optimization effects. Across the 8 publicly deployed frontier models tested, reaching vulnerable code and triggering a crash is routine, but arbitrary code execution against these hardened targets is not — establishing exploit construction against hardened real-world software as an emerging frontier capability rather than a solved problem.

Benchmark Specifications

Benchmarks on Benchgen

The scores below all follow the paper's own official evaluation settings (per Zhipu AI's GLM-5.3 launch blog, which states its evaluation "follows the official evaluation settings": max 300 interaction rounds, average coverage score over all 41 tasks across 3 revisions), so they're directly comparable.

Related Models & Benchmarks

• Models: GLM-5.3 · Kimi K3 · Qwen3.8 Max
• Benchmarks: CyberGym · ExploitGym

Frequently Asked Questions