# WildGuardTest (Response) ![Category](https://img.shields.io/badge/Category-Safety-1f6feb) ![Metric](https://img.shields.io/badge/Metric-F1%20Score-2da44e) ![Saturation](https://img.shields.io/badge/Saturation-Medium-d29922) [![Paper](https://img.shields.io/badge/Paper-arXiv%202406.18495-b31b1b)](https://arxiv.org/abs/2406.18495) [![GitHub](https://img.shields.io/badge/GitHub-allenai%2Fwildguard-181717?logo=github&logoColor=white)](https://github.com/allenai/wildguard) > **Quick answer:** WildGuardTest (Response) measures whether a safety classifier correctly identifies harmful model outputs, using the same 1,725-item test set as WildGuardTest (Prompt) but scoring the response-harm label instead. Response classification is generally the harder task of the two. ## At a Glance **What it tests:** Binary classification of whether an AI model's response to a prompt is harmful, given both the prompt and the response. **Why it matters:** Response moderation must account for context — the same response text can be harmful or benign depending on the prompt it answers, making this a meaningfully different (and often lower-scoring) task than prompt classification. **Known limitations:** English-only; response harm labels have lower inter-annotator agreement (Fleiss κ ≈ 0.50) than prompt harm labels in the original WildGuard study. ## What WildGuardTest (Response) Measures Using the same 1,725-item WildGuardTest split as the prompt-classification task, this benchmark scores the `response_harm_label` field instead of `prompt_harm_label`. A classifier must judge whether the model's reply — not just the user's request — contains harmful content, correctly handling cases like a harmful prompt met with a safe refusal, or a benign prompt met with an unexpectedly harmful completion. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Safety / content moderation | | Metric | F1 score (%) | | Test set size | 1,725 items | | Created by | Han, Rao, Ettinger, Jiang, Lin, Lambert, Choi, Dziri (Allen Institute for AI) | | Paper | [WildGuard (arXiv 2406.18495)](https://arxiv.org/abs/2406.18495) | | GitHub | [allenai/wildguard](https://github.com/allenai/wildguard) | | Dataset | [allenai/wildguardmix](https://huggingface.co/datasets/allenai/wildguardmix) | | License | ODC-BY (dataset), Apache 2.0 (model) | ## How WildGuardTest (Response) Is Scored Each item's ground-truth response-harm label is compared to the classifier's prediction and scored as F1 across the full 1,725-item set. ## State-of-the-Art Results Scores from the Shieldstral model card (Mistral AI, August 2026). F1 (%). | Rank | Model | Score | Source | Date | |---|---|---|---|---| | 1 | [GPT-OSS-Safeguard-20B](/models/openai/gpt-oss-safeguard-20b) | **80.7%** | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | | 2 | [Shieldstral 1.0](/models/mistral-ai/shieldstral-1-0) | 80.4% | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | | 3 | [Qwen3Guard-8B](/models/alibaba/qwen3guard-8b) | 79.6% | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | | 4 | [Nemotron-3.5-Content-Safety-4B](/models/nvidia/nemotron-3-5-content-safety-4b) | 77.6% | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | | 5 | [LlamaGuard-4-12B](/models/meta/llamaguard-4-12b) | 66.8% | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | | 6 | [ShieldGemma-9B](/models/google/shieldgemma-9b) | 34.5% | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | *Scores sourced from Mistral AI's published model card, shown for context. Not Benchgen measurements.* ## WildGuardTest (Response) on Benchgen > No Benchgen results yet — [be the first to run WildGuardTest (Response)](/benchmarks/allen-institute-for-ai/wildguardtest-response). ## WildGuardTest (Response) vs Other Benchmarks | Benchmark | What it tests | Test size | Saturation | |---|---|---|---| | **WildGuardTest (Response)** | Response-harm classification | 1,725 | Medium | | Aegis v2 (Response) | Response classification, 12-category taxonomy | 1,964 | Medium | | HarmBench (Response) | Jailbreak-response harm detection | 400 | Medium | | BeaverTails | QA-pair harm classification | ~700 | Medium | ## Run WildGuardTest (Response) on Your Model Benchgen tracks version-controlled, regression-tested guard-model performance — submit your classifier's response-harm scores to compare against Shieldstral and other guard models. ## Related Models & Benchmarks - Models: [Shieldstral 1.0](/models/mistral-ai/shieldstral-1-0) · [GPT-OSS-Safeguard-20B](/models/openai/gpt-oss-safeguard-20b) · [Qwen3Guard-8B](/models/alibaba/qwen3guard-8b) - Benchmarks: [WildGuardTest (Prompt)](/benchmarks/allen-institute-for-ai/wildguardtest-prompt) · [WildGuardTest (Refusal)](/benchmarks/allen-institute-for-ai/wildguardtest-refusal) · [Aegis v2 (Response)](/benchmarks/nvidia/aegis-v2-response) ## Frequently Asked Questions

What is WildGuardTest (Response)?

It's the response-harm classification task scored on AI2's 1,725-item WildGuardTest set — judging whether a model's reply to a prompt is harmful, as opposed to judging the prompt itself.

Why does response classification score lower than prompt classification?

Response harm depends on context (the prompt-response pair) and has lower human inter-annotator agreement in the original study, making it a harder, noisier signal to fit than prompt-only classification.

Who created WildGuardTest?

The Allen Institute for AI (AI2), introduced in "WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs" (Han et al., NeurIPS 2024).

--- *Last updated 2026-08-12.*

Rank	Model	Score
1	gpt-oss-safeguard-20b	80.7
2	shieldstral-1-0	80.4
3	qwen3guard-8b	79.6
4	nemotron-3-5-content-safety-4b	77.6
5	llamaguard-4-12b	66.8
6	shieldgemma-9b	34.5

WildGuardTest (Response) — Results

WildGuardTest (Response) — Evaluations