Aegis v2 (Response) — Results
| Rank | Model | Score |
|---|---|---|
| 1 | shieldstral-1-0 | 87.2 |
| 2 | qwen3guard-8b | 86.2 |
| 3 | nemotron-3-5-content-safety-4b | 84.9 |
| 4 | gpt-oss-safeguard-20b | 75.2 |
| 5 | llamaguard-4-12b | 64.7 |
| 6 | shieldgemma-9b | 59.7 |
Aegis v2 (Response)
1 phaseActive
NVIDIA's Nemotron Content Safety Dataset V2, response-classification task. 1,964-item test set, 12-category taxonomy. Ghosh et al., 2025.
Overview
Aegis v2 (Response)
Quick answer: Aegis v2 (Response) measures response-harm classification against NVIDIA's 12-category safety taxonomy, using the same 1,964-item held-out test set as Aegis v2 (Prompt) but scoring the LLM's reply rather than the user's request.
At a Glance
What it tests: Whether a classifier correctly flags harmful model responses across NVIDIA's hazard taxonomy, given the prompt-response pair. Why it matters: Response labels in Aegis v2 are hybrid — human-annotated where possible, augmented via an LLM-jury (Mixtral-8x22B, Mistral-NeMo-12B-Instruct, Gemma-2-27B-it) for scale — making this benchmark a useful test of whether a classifier's judgments align with a multi-model consensus process as well as human raters. Known limitations: Responses were generated by a single base model (Mistral-7B-v0.1), so response style/harm patterns may not generalize to responses from more safety-aligned or differently-trained models.
What Aegis v2 (Response) Measures
This task scores the response_label field on the 1,964-item Aegis v2 test set — judging whether the LLM's reply to a (potentially harmful) prompt itself violates the taxonomy, including cases augmented with synthetic refusal data to test whether classifiers correctly recognize safe refusals as non-harmful.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Safety / content moderation |
| Metric | F1 score (%) |
| Test set size | 1,964 items |
| Taxonomy | 12 core + 9 fine-grained hazard categories |
| Created by | Shaona Ghosh, Prasoon Varshney, Makesh Narsimhan Sreedhar, Aishwarya Padmakumar, Traian Rebedea, Jibin Rajan Varghese, Christopher Parisien (NVIDIA) |
| Paper | AEGIS2.0 (NAACL 2025) |
| Dataset | nvidia/Aegis-AI-Content-Safety-Dataset-2.0 |
| License | CC-BY-4.0 |
How Aegis v2 (Response) Is Scored
Each of the 1,964 test items' ground-truth response-harm label is compared against a classifier's prediction, scored as F1.
State-of-the-Art Results
Scores from the Shieldstral model card (Mistral AI, August 2026). F1 (%).
| Rank | Model | Score | Source | Date |
|---|---|---|---|---|
| 1 | Shieldstral 1.0 | 87.2% | Mistral AI model card | 2026-08 |
| 2 | Qwen3Guard-8B | 86.2% | Mistral AI model card | 2026-08 |
| 3 | Nemotron-3.5-Content-Safety-4B | 84.9% | Mistral AI model card | 2026-08 |
| 4 | GPT-OSS-Safeguard-20B | 75.2% | Mistral AI model card | 2026-08 |
| 5 | LlamaGuard-4-12B | 64.7% | Mistral AI model card | 2026-08 |
| 6 | ShieldGemma-9B | 59.7% | Mistral AI model card | 2026-08 |
Scores sourced from Mistral AI's published model card, shown for context. Not Benchgen measurements.
Aegis v2 (Response) on Benchgen
No Benchgen results yet — be the first to run Aegis v2 (Response).
Aegis v2 (Response) vs Other Benchmarks
| Benchmark | What it tests | Test size | Saturation |
|---|---|---|---|
| Aegis v2 (Response) | Fine-grained taxonomy response classification | 1,964 | Medium |
| WildGuardTest (Response) | Response-harm classification | 1,725 | Medium |
| HarmBench (Response) | Adversarial jailbreak response classification | 400 | Medium |
| BeaverTails | QA-pair harm classification | ~700 | Medium |
Run Aegis v2 (Response) on Your Model
Benchgen tracks version-controlled, regression-tested guard-model performance — submit your classifier's Aegis v2 response scores to compare against Shieldstral and other guard models.
Related Models & Benchmarks
- Models: Shieldstral 1.0 · Nemotron-3.5-Content-Safety-4B
- Benchmarks: Aegis v2 (Prompt) · WildGuardTest (Response)
Frequently Asked Questions
What is Aegis v2 (Response)?
It's the response-harm classification task scored on NVIDIA's 1,964-item Aegis v2 (Nemotron Content Safety Dataset V2) test set.How were response labels generated?
Through a hybrid process: human annotation for the base labels, augmented by an LLM-jury (Mixtral-8x22B, Mistral-NeMo-12B-Instruct, Gemma-2-27B-it) for additional coverage, plus synthetic refusal data.Who created Aegis v2?
NVIDIA's NeMo Guardrails team (Ghosh et al.), published at NAACL 2025.Last updated 2026-08-12.