# ExploitBench ![Category](https://img.shields.io/badge/Category-Agent-1f6feb) ![Metric](https://img.shields.io/badge/Metric-%25%20Coverage-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-41-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-Low-d29922) ![Created](https://img.shields.io/badge/Created-May%202026-555) [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/abs/2605.14153) [![GitHub](https://img.shields.io/badge/GitHub-exploitbench-181717?logo=github&logoColor=white)](https://github.com/exploitbench/exploitbench) [![Site](https://img.shields.io/badge/Site-exploitbench.ai-0969da)](https://exploitbench.ai/) > **Quick answer:** ExploitBench treats exploitation as a ladder, not a binary crash/no-crash outcome. It decomposes exploiting a real vulnerability into 16 measurable capability flags — from reaching vulnerable code, through triggering a crash, building sandbox primitives and arbitrary read/write, up to full arbitrary code execution — verified by deterministic oracles across 41 real, hardened V8 JavaScript engine bugs. ## At a Glance **What it tests:** How far an AI agent can climb a graded exploitation capability ladder against real, production-hardened vulnerabilities in V8 (the JS/WASM engine behind Chrome, Edge, and Node.js) — not just whether it can crash the target. **Why it matters:** Prior benchmarks collapse exploitation into a single binary success/failure signal, hiding exactly where models struggle. ExploitBench's 16-flag ladder shows that reaching and crashing vulnerable code is comparatively routine for frontier models, while arbitrary code execution against hardened targets remains an emerging, largely unsolved capability. **Known limitations:** Narrow domain — 41 tasks, all V8-specific. The authors note a sharp split between publicly deployed frontier models (where full code execution is rare) and an unnamed private/internal model (which reaches it roughly half the time), suggesting public model results understate the frontier's true capability ceiling. ## What ExploitBench Measures Each of ExploitBench's 41 instances is a real V8 bug, chosen specifically because V8 is both widely deployed and heavily exploitation-hardened. Rather than scoring a single "did it work" outcome, the benchmark decomposes exploitation into 16 measurable flags spanning five tiers: coverage, bug triggering, V8-internal primitives, general memory-corruption primitives, and arbitrary code execution. Each flag is verified by a deterministic oracle — using per-run randomized challenge-response checks for primitives, differential execution against ground-truth binaries to measure progress, and a signal-handler proof for code execution — so results can't be gamed by a lucky crash. The paper reports results across three evaluation arms: `` as the primary measure of model-environment capability, `` to test whether targeted feedback shifts outcomes, and `` as an ablation swapping in a vendor's native CLI harness to check for vendor-side optimization effects. Across the 8 publicly deployed frontier models tested, reaching vulnerable code and triggering a crash is routine, but arbitrary code execution against these hardened targets is not — establishing exploit construction against hardened real-world software as an emerging frontier capability rather than a solved problem. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Cybersecurity / Agent | | Metric | Average % coverage score (share of the 16-flag capability ladder achieved) across 3 revisions per task | | Number of tasks | 41 (all real V8 bugs) | | Primary evaluation arm | `` | | Saturation | Low | | Created by | Seunghyun Lee, David Brumley | | Paper | [arXiv:2605.14153](https://arxiv.org/abs/2605.14153) | ## Benchmarks on Benchgen The scores below all follow the paper's own official evaluation settings (per Zhipu AI's GLM-5.3 launch blog, which states its evaluation "follows the official evaluation settings": max 300 interaction rounds, average coverage score over all 41 tasks across 3 revisions), so they're directly comparable. | Model | Score | Source | Date | |---|---|---|---| | [GLM-5.3](/models/zhipuai/glm-5-3) | 54.4% | Zhipu AI GLM-5.3 launch blog | 2026-08 | | [Kimi K3](/models/moonshot-ai/kimi-k3) | 32.2% | Zhipu AI GLM-5.3 launch blog | 2026-08 | | [Qwen3.8 Max](/models/alibaba/qwen3-8-max) | 28.8% | Zhipu AI GLM-5.3 launch blog | 2026-08 | | [GLM-5.2](/models/zhipu-ai/glm-5-2) | 24.4% | Zhipu AI GLM-5.3 launch blog | 2026-08 | ## Related Models & Benchmarks - Models: [GLM-5.3](/models/zhipuai/glm-5-3) · [Kimi K3](/models/moonshot-ai/kimi-k3) · [Qwen3.8 Max](/models/alibaba/qwen3-8-max) - Benchmarks: [CyberGym](/benchmarks/zhun-wang-tianneng-shi-et-al-uc-berkeley/cybergym) · [ExploitGym](/benchmarks/zhun-wang-nico-schiller-et-al-uc-berkeley/exploitgym) ## Frequently Asked Questions

What is ExploitBench?

A cybersecurity benchmark that grades AI agents on a 16-flag capability ladder for exploiting real, hardened V8 JavaScript engine vulnerabilities — rather than scoring exploitation as a simple pass/fail.

What does a good score look like?

Scores in this benchmark reflect average coverage of the 16-flag ladder. Published frontier-model scores are currently in the 20-55% range, with full arbitrary code execution against hardened targets still rare for publicly deployed models.

Who created ExploitBench?

Seunghyun Lee and David Brumley, published on arXiv in May 2026, with code at github.com/exploitbench/exploitbench and a project site at exploitbench.ai.

Rank	Model	Score
1	gpt-6-astra	100
2	glm-5-3	54.4
3	mimo-v2-6-pro-rl	47.9
4	kimi-k3	32.2
5	qwen3-8-max	28.8
6	mimo-v2-6-flash-rl	25.3
7	glm-5-2	24.4

ExploitBench — Results

ExploitBench — Evaluations