WildGuardTest (Refusal) — Results
| Rank | Model | Score |
|---|---|---|
| 1 | gpt-oss-safeguard-20b | 93.9 |
| 2 | shieldstral-1-0 | 90.3 |
| 3 | qwen3guard-8b | 90.1 |
| 4 | wildguard-7b | 88.6 |
| 5 | polyguard-qwen-7b | 87.5 |
WildGuardTest (Refusal)
1 phaseActive
AI2's held-out test set for refusal-vs-compliance classification. 1,725 human-annotated items, F1 metric. From the WildGuard paper (Han et al., 2024).
Overview
WildGuardTest (Refusal)
Quick answer: WildGuardTest (Refusal) measures whether a classifier can correctly detect if a model refused vs. complied with a request, using the same 1,725-item WildGuardTest set. This is useful for tracking over-refusal and under-refusal behavior independent of harm labels.
At a Glance
What it tests: Binary classification of whether a model's response is a refusal or a compliance, regardless of whether the underlying prompt was harmful. Why it matters: Refusal detection is a building block for measuring both under-refusal (models complying with harmful requests) and over-refusal (models refusing benign requests), a known failure mode this task can help surface. Known limitations: English-only; refusal labels achieved the highest inter-annotator agreement of the three WildGuard tasks (Fleiss κ ≈ 0.72), making this the most reliable of the three labels.
What WildGuardTest (Refusal) Measures
This task scores the response_refusal_label field on the 1,725-item WildGuardTest set — classifying each response as "refusal" or "compliance" independent of harm. Combined with prompt- and response-harm labels, refusal detection lets a full moderation pipeline distinguish, for example, a harmful prompt correctly refused (safe outcome) from a harmful prompt answered (unsafe outcome) or a benign prompt incorrectly refused (over-refusal, a UX cost).
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Safety / content moderation |
| Metric | F1 score (%) |
| Test set size | 1,725 items |
| Created by | Han, Rao, Ettinger, Jiang, Lin, Lambert, Choi, Dziri (Allen Institute for AI) |
| Paper | WildGuard (arXiv 2406.18495) |
| GitHub | allenai/wildguard |
| Dataset | allenai/wildguardmix |
| License | ODC-BY (dataset), Apache 2.0 (model) |
How WildGuardTest (Refusal) Is Scored
Each item's ground-truth refusal label is compared against the classifier's prediction and scored as F1 across the 1,725-item set.
State-of-the-Art Results
Scores from the Shieldstral model card (Mistral AI, August 2026). F1 (%).
| Rank | Model | Score | Source | Date |
|---|---|---|---|---|
| 1 | GPT-OSS-Safeguard-20B | 93.9% | Mistral AI model card | 2026-08 |
| 2 | Shieldstral 1.0 | 90.3% | Mistral AI model card | 2026-08 |
| 3 | Qwen3Guard-8B | 90.1% | Mistral AI model card | 2026-08 |
| 4 | WildGuard-7B | 88.6% | Mistral AI model card | 2026-08 |
| 5 | PolyGuard-Qwen-7B | 87.5% | Mistral AI model card | 2026-08 |
Scores sourced from Mistral AI's published model card, shown for context. Not Benchgen measurements. LlamaGuard-4, ShieldGemma, and Nemotron-3.5-Content-Safety variants do not report refusal-detection scores in this comparison.
WildGuardTest (Refusal) on Benchgen
No Benchgen results yet — be the first to run WildGuardTest (Refusal).
WildGuardTest (Refusal) vs Other Benchmarks
| Benchmark | What it tests | Test size | Saturation |
|---|---|---|---|
| WildGuardTest (Refusal) | Refusal vs. compliance detection | 1,725 | Medium |
| XSTest (Refusal) | Over-refusal on safe-but-sensitive prompts | 450 | Medium |
| PolyGuard (Refusal) | Multilingual refusal detection | 29,325 | Medium |
Run WildGuardTest (Refusal) on Your Model
Benchgen tracks version-controlled, regression-tested guard-model performance — submit your classifier's refusal-detection scores to compare against Shieldstral and other guard models.
Related Models & Benchmarks
- Models: Shieldstral 1.0 · GPT-OSS-Safeguard-20B · Qwen3Guard-8B
- Benchmarks: WildGuardTest (Prompt) · WildGuardTest (Response) · XSTest (Refusal)
Frequently Asked Questions
What is WildGuardTest (Refusal)?
It's the refusal-vs-compliance classification task scored on AI2's 1,725-item WildGuardTest set, independent of whether the underlying content was harmful.Why is refusal detection useful separately from harm detection?
It lets teams measure over-refusal (declining safe requests) and under-refusal (complying with harmful requests) as distinct failure modes, rather than conflating them into a single harm score.Who created WildGuardTest?
The Allen Institute for AI (AI2), introduced in "WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs" (Han et al., NeurIPS 2024).Last updated 2026-08-12.