ExploitGym

ExploitGym

Quick answer: ExploitGym is a UC Berkeley cybersecurity agent benchmark that goes one step past vulnerability discovery: given a program input that already triggers a known vulnerability, the agent must progressively extend it into a working exploit. It spans real-world vulnerabilities in userspace programs, Google's V8 JavaScript engine, and the Linux kernel, under varying security protections.

At a Glance

What it tests: Whether an AI agent can turn a bare vulnerability trigger into a concrete security impact (e.g., unauthorized file access or code execution) — the "last mile" of offensive security capability, requiring low-level program reasoning and sustained progress over long horizons.

Why it matters: Exploitation is dual-use — the same capability that lets defenders validate and prioritize real risk also lowers the barrier to offense. ExploitGym is one of the few benchmarks measuring this capability directly rather than treating a mere crash as "success."

Known limitations: Reported scores can vary significantly by harness/time-budget protocol, since exploitation runtime varies with per-model inference speed. Cross-source scores using different time budgets or harnesses are not directly comparable — check the protocol before comparing numbers across sources.

What ExploitGym Measures

Each ExploitGym instance is built from a real-world vulnerability with a known proof-of-vulnerability (PoV) input already provided — the crash-triggering input is a given, not something the agent has to find. The agent's job is to iteratively extend that PoV into a working exploit that achieves a concrete impact, with security protections (e.g., ASLR, sandboxing) varied per instance to isolate their effect on agent performance. All instances run in reproducible, containerized environments.

The benchmark's own evaluation found that while exploitation remains hard, frontier models can already succeed on a non-trivial fraction of instances — and that even with common defenses enabled, success rates stay meaningfully above zero, a signal the authors treat as an early warning about the offensive potential of increasingly capable agents.

Benchmark Specifications

State-of-the-Art Results

Scores from the original ExploitGym paper (Wang et al., May 2026), using the paper's own default evaluation protocol — not time-budgeted. Note: Claude Mythos Preview's public access has since been suspended (per its Benchgen model page); this result predates that suspension.

Benchmarks on Benchgen

The scores below use a different, TPS-normalized time-budget protocol (2h / 6h per instance, rescaled by each model's tokens-per-second rate) reported in Zhipu AI's GLM-5.3 launch blog — evaluated via Claude Code 2.1.207 across 869 instances. Because this protocol differs from the original paper's (see State-of-the-Art Results above), these four scores are directly comparable to each other but not to the table above. The score column below reports the 6-hour-budget count; 2-hour-budget counts are shown for reference only.

Related Models & Benchmarks

• Models: GLM-5.3 · Kimi K3 · Qwen3.8 Max · GPT-5.5
• Benchmarks: CyberGym · ExploitBench

Frequently Asked Questions