# WildGuardTest (Refusal) ![Category](https://img.shields.io/badge/Category-Safety-1f6feb) ![Metric](https://img.shields.io/badge/Metric-F1%20Score-2da44e) ![Saturation](https://img.shields.io/badge/Saturation-Medium-d29922) [![Paper](https://img.shields.io/badge/Paper-arXiv%202406.18495-b31b1b)](https://arxiv.org/abs/2406.18495) [![GitHub](https://img.shields.io/badge/GitHub-allenai%2Fwildguard-181717?logo=github&logoColor=white)](https://github.com/allenai/wildguard) > **Quick answer:** WildGuardTest (Refusal) measures whether a classifier can correctly detect if a model refused vs. complied with a request, using the same 1,725-item WildGuardTest set. This is useful for tracking over-refusal and under-refusal behavior independent of harm labels. ## At a Glance **What it tests:** Binary classification of whether a model's response is a refusal or a compliance, regardless of whether the underlying prompt was harmful. **Why it matters:** Refusal detection is a building block for measuring both under-refusal (models complying with harmful requests) and over-refusal (models refusing benign requests), a known failure mode this task can help surface. **Known limitations:** English-only; refusal labels achieved the highest inter-annotator agreement of the three WildGuard tasks (Fleiss κ ≈ 0.72), making this the most reliable of the three labels. ## What WildGuardTest (Refusal) Measures This task scores the `response_refusal_label` field on the 1,725-item WildGuardTest set — classifying each response as "refusal" or "compliance" independent of harm. Combined with prompt- and response-harm labels, refusal detection lets a full moderation pipeline distinguish, for example, a harmful prompt correctly refused (safe outcome) from a harmful prompt answered (unsafe outcome) or a benign prompt incorrectly refused (over-refusal, a UX cost). ## Benchmark Specifications | Field | Value | |---|---| | Task category | Safety / content moderation | | Metric | F1 score (%) | | Test set size | 1,725 items | | Created by | Han, Rao, Ettinger, Jiang, Lin, Lambert, Choi, Dziri (Allen Institute for AI) | | Paper | [WildGuard (arXiv 2406.18495)](https://arxiv.org/abs/2406.18495) | | GitHub | [allenai/wildguard](https://github.com/allenai/wildguard) | | Dataset | [allenai/wildguardmix](https://huggingface.co/datasets/allenai/wildguardmix) | | License | ODC-BY (dataset), Apache 2.0 (model) | ## How WildGuardTest (Refusal) Is Scored Each item's ground-truth refusal label is compared against the classifier's prediction and scored as F1 across the 1,725-item set. ## State-of-the-Art Results Scores from the Shieldstral model card (Mistral AI, August 2026). F1 (%). | Rank | Model | Score | Source | Date | |---|---|---|---|---| | 1 | [GPT-OSS-Safeguard-20B](/models/openai/gpt-oss-safeguard-20b) | **93.9%** | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | | 2 | [Shieldstral 1.0](/models/mistral-ai/shieldstral-1-0) | 90.3% | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | | 3 | [Qwen3Guard-8B](/models/alibaba/qwen3guard-8b) | 90.1% | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | | 4 | [WildGuard-7B](/models/allen-institute-for-ai/wildguard-7b) | 88.6% | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | | 5 | [PolyGuard-Qwen-7B](/models/toxicityprompts/polyguard-qwen-7b) | 87.5% | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | *Scores sourced from Mistral AI's published model card, shown for context. Not Benchgen measurements. LlamaGuard-4, ShieldGemma, and Nemotron-3.5-Content-Safety variants do not report refusal-detection scores in this comparison.* ## WildGuardTest (Refusal) on Benchgen > No Benchgen results yet — [be the first to run WildGuardTest (Refusal)](/benchmarks/allen-institute-for-ai/wildguardtest-refusal). ## WildGuardTest (Refusal) vs Other Benchmarks | Benchmark | What it tests | Test size | Saturation | |---|---|---|---| | **WildGuardTest (Refusal)** | Refusal vs. compliance detection | 1,725 | Medium | | XSTest (Refusal) | Over-refusal on safe-but-sensitive prompts | 450 | Medium | | PolyGuard (Refusal) | Multilingual refusal detection | 29,325 | Medium | ## Run WildGuardTest (Refusal) on Your Model Benchgen tracks version-controlled, regression-tested guard-model performance — submit your classifier's refusal-detection scores to compare against Shieldstral and other guard models. ## Related Models & Benchmarks - Models: [Shieldstral 1.0](/models/mistral-ai/shieldstral-1-0) · [GPT-OSS-Safeguard-20B](/models/openai/gpt-oss-safeguard-20b) · [Qwen3Guard-8B](/models/alibaba/qwen3guard-8b) - Benchmarks: [WildGuardTest (Prompt)](/benchmarks/allen-institute-for-ai/wildguardtest-prompt) · [WildGuardTest (Response)](/benchmarks/allen-institute-for-ai/wildguardtest-response) · [XSTest (Refusal)](/benchmarks/independent-research/xstest-refusal) ## Frequently Asked Questions

What is WildGuardTest (Refusal)?

It's the refusal-vs-compliance classification task scored on AI2's 1,725-item WildGuardTest set, independent of whether the underlying content was harmful.

Why is refusal detection useful separately from harm detection?

It lets teams measure over-refusal (declining safe requests) and under-refusal (complying with harmful requests) as distinct failure modes, rather than conflating them into a single harm score.

Who created WildGuardTest?

The Allen Institute for AI (AI2), introduced in "WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs" (Han et al., NeurIPS 2024).

--- *Last updated 2026-08-12.*

Rank	Model	Score
1	gpt-oss-safeguard-20b	93.9
2	shieldstral-1-0	90.3
3	qwen3guard-8b	90.1
4	wildguard-7b	88.6
5	polyguard-qwen-7b	87.5

WildGuardTest (Refusal) — Results

WildGuardTest (Refusal) — Evaluations