VLoc Bench

VLoc Bench

Quick answer: VLoc Bench (Vulnerability Localization Benchmark) is a 500-task agentic security benchmark created by Cisco Foundation AI that tests whether AI models can identify vulnerable source files in real codebases using only a terminal interface and a CWE description — no file hints, no advisory text. The current SOTA is 0.221 File F1 (GPT-5.5 / Antares-3B), with Antares 1B reaching 0.209 at 1B parameters — outperforming GLM-5.2 (753B), Gemini 3 Pro, and GPT-5 Mini.

At a Glance

What it tests: A model operating as a terminal agent — issuing shell commands (grep, find, cat) — must navigate an unfamiliar codebase and submit a ranked list of files it believes contain a given CWE-category vulnerability. No RAG, no vector index: pure agentic exploration within a 15-command budget.

Why it matters: VLoc Bench is the first standardised benchmark focused specifically on repository-level vulnerability localization — the expensive first step of any security triage workflow. It separates models genuinely capable of security-aware code reasoning from those that merely score well on general coding benchmarks.

Known limitations: The 15-command budget and task-level F1 metric penalise large repositories. Models are evaluated at temperature 0.3, and scores are averaged across 3 runs. The benchmark covers 147 CWE categories — performance varies significantly by CWE type, with pattern-matchable vulnerabilities (e.g. CWE-843, CWE-1321) scoring higher than semantic ones (e.g. CWE-732, CWE-667).

What VLoc Bench Measures

Each of the 500 tasks presents a model with a real repository snapshot (reconstructed at the pre-fix commit) and a generic CWE category description — nothing more. The model must explore the codebase through a Unix terminal, issue up to 15 shell commands, and then call submit_vulnerable_files with a list of file paths it believes are affected.

The benchmark draws from 290 unique real-world repositories across 6 package ecosystems, spanning 147 CWE categories. 78% of tasks have assigned CVE identifiers. Ground truth consists of the implementation files modified in the actual security-fix pull request (excluding tests, docs, and configuration files).

Scoring uses task-level File F1 — the harmonic mean of file-level precision and recall for each task — macro-averaged across all 500 tasks and then averaged over 3 independent runs. A model that submits no files or fails to call a valid terminal action scores zero for that task.

The benchmark was introduced alongside the Antares model family by Cisco Foundation AI to fill the gap between general SWE-bench-style coding benchmarks and the specific challenge of security-first code localization.

Benchmark Specifications

How VLoc Bench Is Scored

For each of the 500 tasks, the model's submitted file list is compared against the ground-truth set of vulnerable implementation files. Precision is the fraction of submitted files that are in the ground truth; recall is the fraction of ground-truth files that were submitted. Task-level File F1 is the harmonic mean of task precision and recall. A task with no valid submission scores zero across all metrics.

Final scores are the macro-average of task-level F1 across all 500 tasks, then averaged across 3 independent runs. This means every task counts equally regardless of how many ground-truth files it contains.

State-of-the-Art Results

Scores represent File F1 (0–1 scale). All models evaluated using the same 15-command agent loop, temperature 0.3, top-p 1.0, averaged over 3 runs.

Source: Antares technical report, July 2026. Antares 3B (GRPO: 0.223) is listed in the paper but not yet publicly released.

Last updated 2026-07-24.