# ExploitGym ![Category](https://img.shields.io/badge/Category-Agent-1f6feb) ![Metric](https://img.shields.io/badge/Metric-%23%20Exploited-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-869-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-Low-d29922) ![Created](https://img.shields.io/badge/Created-May%202026-555) [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/abs/2605.11086) [![GitHub](https://img.shields.io/badge/GitHub-exploitgym-181717?logo=github&logoColor=white)](https://github.com/sunblaze-ucb/exploitgym) > **Quick answer:** ExploitGym is a UC Berkeley cybersecurity agent benchmark that goes one step past vulnerability discovery: given a program input that already triggers a known vulnerability, the agent must progressively extend it into a working exploit. It spans real-world vulnerabilities in userspace programs, Google's V8 JavaScript engine, and the Linux kernel, under varying security protections. ## At a Glance **What it tests:** Whether an AI agent can turn a bare vulnerability trigger into a concrete security impact (e.g., unauthorized file access or code execution) — the "last mile" of offensive security capability, requiring low-level program reasoning and sustained progress over long horizons. **Why it matters:** Exploitation is dual-use — the same capability that lets defenders validate and prioritize real risk also lowers the barrier to offense. ExploitGym is one of the few benchmarks measuring this capability directly rather than treating a mere crash as "success." **Known limitations:** Reported scores can vary significantly by harness/time-budget protocol, since exploitation runtime varies with per-model inference speed. Cross-source scores using different time budgets or harnesses are not directly comparable — check the protocol before comparing numbers across sources. ## What ExploitGym Measures Each ExploitGym instance is built from a real-world vulnerability with a known proof-of-vulnerability (PoV) input already provided — the crash-triggering input is a given, not something the agent has to find. The agent's job is to iteratively extend that PoV into a working exploit that achieves a concrete impact, with security protections (e.g., ASLR, sandboxing) varied per instance to isolate their effect on agent performance. All instances run in reproducible, containerized environments. The benchmark's own evaluation found that while exploitation remains hard, frontier models can already succeed on a non-trivial fraction of instances — and that even with common defenses enabled, success rates stay meaningfully above zero, a signal the authors treat as an early warning about the offensive potential of increasingly capable agents. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Cybersecurity / Agent | | Metric | # instances successfully exploited (raw count; time-budget/harness varies by source — see note below) | | Number of tasks | 869 (current release; original paper evaluated 898) | | Domains | Userspace programs, Google V8 JS engine, Linux kernel | | Saturation | Low | | Created by | Zhun Wang, Nico Schiller, Hongwei Li et al. (UC Berkeley) | | Paper | [arXiv:2605.11086](https://arxiv.org/abs/2605.11086) | ## State-of-the-Art Results | Rank | Model | Score | Source | Date | |---|---|---|---|---| | 1 | [Claude Mythos Preview](/models/anthropic/claude-mythos-preview) | 157 / 898 | Original ExploitGym paper (unbudgeted protocol) | 2026-05 | | 2 | [GPT-5.5](/models/openai/gpt-5-5) | 120 / 898 | Original ExploitGym paper (unbudgeted protocol) | 2026-05 | *Scores from the original ExploitGym paper (Wang et al., May 2026), using the paper's own default evaluation protocol — not time-budgeted. Note: Claude Mythos Preview's public access has since been suspended (per its Benchgen model page); this result predates that suspension.* ## Benchmarks on Benchgen The scores below use a different, TPS-normalized time-budget protocol (2h / 6h per instance, rescaled by each model's tokens-per-second rate) reported in Zhipu AI's GLM-5.3 launch blog — evaluated via Claude Code 2.1.207 across 869 instances. Because this protocol differs from the original paper's (see State-of-the-Art Results above), these four scores are directly comparable *to each other* but not to the table above. The `score` column below reports the 6-hour-budget count; 2-hour-budget counts are shown for reference only. | Model | Score (6h budget) | 2h budget (reference) | Source | Date | |---|---|---|---|---| | [GLM-5.3](/models/zhipuai/glm-5-3) | 130 | 105 | Zhipu AI GLM-5.3 launch blog | 2026-08 | | [Kimi K3](/models/moonshot-ai/kimi-k3) | 70 | 36 | Zhipu AI GLM-5.3 launch blog | 2026-08 | | [GLM-5.2](/models/zhipu-ai/glm-5-2) | 39 | 29 | Zhipu AI GLM-5.3 launch blog | 2026-08 | | [Qwen3.8 Max](/models/alibaba/qwen3-8-max) | 26 | 14 | Zhipu AI GLM-5.3 launch blog | 2026-08 | ## Related Models & Benchmarks - Models: [GLM-5.3](/models/zhipuai/glm-5-3) · [Kimi K3](/models/moonshot-ai/kimi-k3) · [Qwen3.8 Max](/models/alibaba/qwen3-8-max) · [GPT-5.5](/models/openai/gpt-5-5) - Benchmarks: [CyberGym](/benchmarks/zhun-wang-tianneng-shi-et-al-uc-berkeley/cybergym) · [ExploitBench](/benchmarks/seunghyun-lee-david-brumley/exploitbench) ## Frequently Asked Questions

What is ExploitGym?

A UC Berkeley cybersecurity agent benchmark that gives an agent a working vulnerability-triggering input and asks it to extend that into a full exploit, across 869 real-world instances spanning userspace software, V8, and the Linux kernel.

What does a good score look like?

It depends heavily on the evaluation protocol/time budget used — always check the source's methodology before comparing scores across different reports.

Who created ExploitGym?

Zhun Wang, Nico Schiller and co-authors at UC Berkeley (the same lab behind CyberGym), published on arXiv in May 2026.

Rank	Model	Score
1	glm-5-3	130
2	kimi-k3	70
3	gpt-6-astra	42.4
4	glm-5-2	39
5	qwen3-8-max	26
6	deepseek-v4-1-flash	15.3
7	deepseek-v4-1-flash	15.3

ExploitGym — Results

ExploitGym — Evaluations