CyberGym

CyberGym

Quick answer: CyberGym is a large-scale cybersecurity agent benchmark from UC Berkeley featuring 1,507 real-world vulnerability reproduction tasks sourced from 188 OSS-Fuzz-monitored software projects. Agents are evaluated on their ability to generate a working proof-of-concept (PoC) that reproduces a known vulnerability given only a text description and the unpatched codebase.

At a Glance

What it tests: Whether an AI agent can reason over large codebases and generate a working PoC exploit for a real-world vulnerability, given only a vulnerability description and the unpatched source code.

Why it matters: Vulnerability reproduction is a direct proxy for real-world offensive security capability — high scores on CyberGym correlate with agents' ability to autonomously discover novel zero-day vulnerabilities in production software.

Known limitations: Benchmark evaluates Level 1 difficulty (description + codebase only); harder inputs like stack traces or ground-truth patches dramatically increase success rates. Current top scores remain low (~20–80%), signalling the benchmark is far from saturation.

What CyberGym Measures

CyberGym collects 1,507 instances from vulnerabilities discovered and patched across 188 widely-distributed software projects continuously fuzzed by OSS-Fuzz. Each instance freezes the repository at its pre-patch commit state. An agent receives the vulnerability's text description and the full unpatched codebase, then iteratively generates and refines a PoC test based on execution feedback. Success is confirmed when the PoC triggers a crash on the pre-patch version but cleanly passes on the post-patch version — ensuring the PoC targets the actual vulnerability rather than an unrelated defect.

Beyond static benchmarking, CyberGym has been used to conduct open-ended vulnerability discovery: agents running against the latest codebases of 431 OSS-Fuzz projects uncovered 34 previously unknown zero-day vulnerabilities and flagged 18 historically incomplete patches. This demonstrates a strong correlation between CyberGym performance and real-world offensive security impact.

The benchmark defines four difficulty levels based on input richness — from Level 0 (no description) through Level 3 (ground-truth patch provided). The primary leaderboard uses Level 1, which provides only a textual vulnerability description alongside the codebase. Top models achieve success rates in the 20–80% range on Level 1, with PoC length being the strongest predictor of task difficulty.

Benchmark Specifications

State-of-the-Art Results

Scores from the ByteDance Seed 2.1 technical report (June 2026).