# Aegis v2 (Response) ![Category](https://img.shields.io/badge/Category-Safety-1f6feb) ![Metric](https://img.shields.io/badge/Metric-F1%20Score-2da44e) ![Saturation](https://img.shields.io/badge/Saturation-Medium-d29922) [![Paper](https://img.shields.io/badge/Paper-arXiv%202501.09004-b31b1b)](https://arxiv.org/abs/2501.09004) > **Quick answer:** Aegis v2 (Response) measures response-harm classification against NVIDIA's 12-category safety taxonomy, using the same 1,964-item held-out test set as Aegis v2 (Prompt) but scoring the LLM's reply rather than the user's request. ## At a Glance **What it tests:** Whether a classifier correctly flags harmful model responses across NVIDIA's hazard taxonomy, given the prompt-response pair. **Why it matters:** Response labels in Aegis v2 are hybrid — human-annotated where possible, augmented via an LLM-jury (Mixtral-8x22B, Mistral-NeMo-12B-Instruct, Gemma-2-27B-it) for scale — making this benchmark a useful test of whether a classifier's judgments align with a multi-model consensus process as well as human raters. **Known limitations:** Responses were generated by a single base model (Mistral-7B-v0.1), so response style/harm patterns may not generalize to responses from more safety-aligned or differently-trained models. ## What Aegis v2 (Response) Measures This task scores the `response_label` field on the 1,964-item Aegis v2 test set — judging whether the LLM's reply to a (potentially harmful) prompt itself violates the taxonomy, including cases augmented with synthetic refusal data to test whether classifiers correctly recognize safe refusals as non-harmful. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Safety / content moderation | | Metric | F1 score (%) | | Test set size | 1,964 items | | Taxonomy | 12 core + 9 fine-grained hazard categories | | Created by | Shaona Ghosh, Prasoon Varshney, Makesh Narsimhan Sreedhar, Aishwarya Padmakumar, Traian Rebedea, Jibin Rajan Varghese, Christopher Parisien (NVIDIA) | | Paper | [AEGIS2.0 (NAACL 2025)](https://arxiv.org/abs/2501.09004) | | Dataset | [nvidia/Aegis-AI-Content-Safety-Dataset-2.0](https://huggingface.co/datasets/nvidia/Aegis-AI-Content-Safety-Dataset-2.0) | | License | CC-BY-4.0 | ## How Aegis v2 (Response) Is Scored Each of the 1,964 test items' ground-truth response-harm label is compared against a classifier's prediction, scored as F1. ## State-of-the-Art Results Scores from the Shieldstral model card (Mistral AI, August 2026). F1 (%). | Rank | Model | Score | Source | Date | |---|---|---|---|---| | 1 | [Shieldstral 1.0](/models/mistral-ai/shieldstral-1-0) | **87.2%** | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | | 2 | [Qwen3Guard-8B](/models/alibaba/qwen3guard-8b) | 86.2% | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | | 3 | [Nemotron-3.5-Content-Safety-4B](/models/nvidia/nemotron-3-5-content-safety-4b) | 84.9% | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | | 4 | [GPT-OSS-Safeguard-20B](/models/openai/gpt-oss-safeguard-20b) | 75.2% | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | | 5 | [LlamaGuard-4-12B](/models/meta/llamaguard-4-12b) | 64.7% | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | | 6 | [ShieldGemma-9B](/models/google/shieldgemma-9b) | 59.7% | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | *Scores sourced from Mistral AI's published model card, shown for context. Not Benchgen measurements.* ## Aegis v2 (Response) on Benchgen > No Benchgen results yet — [be the first to run Aegis v2 (Response)](/benchmarks/nvidia/aegis-v2-response). ## Aegis v2 (Response) vs Other Benchmarks | Benchmark | What it tests | Test size | Saturation | |---|---|---|---| | **Aegis v2 (Response)** | Fine-grained taxonomy response classification | 1,964 | Medium | | WildGuardTest (Response) | Response-harm classification | 1,725 | Medium | | HarmBench (Response) | Adversarial jailbreak response classification | 400 | Medium | | BeaverTails | QA-pair harm classification | ~700 | Medium | ## Run Aegis v2 (Response) on Your Model Benchgen tracks version-controlled, regression-tested guard-model performance — submit your classifier's Aegis v2 response scores to compare against Shieldstral and other guard models. ## Related Models & Benchmarks - Models: [Shieldstral 1.0](/models/mistral-ai/shieldstral-1-0) · [Nemotron-3.5-Content-Safety-4B](/models/nvidia/nemotron-3-5-content-safety-4b) - Benchmarks: [Aegis v2 (Prompt)](/benchmarks/nvidia/aegis-v2-prompt) · [WildGuardTest (Response)](/benchmarks/allen-institute-for-ai/wildguardtest-response) ## Frequently Asked Questions

What is Aegis v2 (Response)?

It's the response-harm classification task scored on NVIDIA's 1,964-item Aegis v2 (Nemotron Content Safety Dataset V2) test set.

How were response labels generated?

Through a hybrid process: human annotation for the base labels, augmented by an LLM-jury (Mixtral-8x22B, Mistral-NeMo-12B-Instruct, Gemma-2-27B-it) for additional coverage, plus synthetic refusal data.

Who created Aegis v2?

NVIDIA's NeMo Guardrails team (Ghosh et al.), published at NAACL 2025.

--- *Last updated 2026-08-12.*

Rank	Model	Score
1	shieldstral-1-0	87.2
2	qwen3guard-8b	86.2
3	nemotron-3-5-content-safety-4b	84.9
4	gpt-oss-safeguard-20b	75.2
5	llamaguard-4-12b	64.7
6	shieldgemma-9b	59.7

Aegis v2 (Response) — Results

Aegis v2 (Response) — Evaluations