XSTest (Refusal) — Results
| Rank | Model | Score |
|---|---|---|
| 1 | gpt-oss-safeguard-20b | 94.9 |
| 2 | shieldstral-1-0 | 94.6 |
| 3 | qwen3guard-8b | 92.6 |
| 4 | wildguard-7b | 92.5 |
| 5 | polyguard-qwen-7b | 82.1 |
XSTest (Refusal)
1 phaseActive
A 450-prompt test suite for detecting over-refusal (exaggerated safety) behavior. Röttger et al., 2023.
Overview
XSTest (Refusal)
Quick answer: XSTest (Refusal) measures whether a classifier can correctly detect over-refusal — a target model declining to answer a genuinely safe prompt because it superficially resembles an unsafe one. It uses the same 450-item XSTest suite as the harm-classification task but scores refusal detection on model completions specifically.
At a Glance
What it tests: Whether a classifier can distinguish a genuine refusal from a compliant answer, applied to completions generated on XSTest's 250 safe / 200 unsafe prompt set. Why it matters: Over-refusal is a real UX cost in production systems — models that decline safe requests because of surface-level similarity to unsafe ones frustrate users. A refusal classifier needs to catch this accurately to support over-refusal monitoring. Known limitations: Small test set (450 items); results depend on which target model's completions were used to generate the refusal/compliance labels being evaluated.
What XSTest (Refusal) Measures
This task scores whether a classifier correctly labels a model's response to an XSTest prompt as a refusal or a compliance. Applied across both the safe and unsafe halves of the set, it surfaces two distinct failure modes: refusing a safe prompt (over-refusal) and complying with an unsafe prompt (under-refusal) — with the safe-prompt half being the more diagnostic test for exaggerated safety specifically.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Safety / refusal detection |
| Metric | F1 score (%) |
| Test set size | 450 (250 safe + 200 unsafe) |
| Created by | Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, Dirk Hovy |
| Paper | XSTest (arXiv 2308.01263) |
| GitHub | paul-rottger/exaggerated-safety |
| Dataset | walledai/XSTest |
| License | CC-BY-4.0 (prompts) |
How XSTest (Refusal) Is Scored
Each response's ground-truth refusal/compliance label is compared against a classifier's prediction and scored as F1 across the 450-item set.
State-of-the-Art Results
Scores from the Shieldstral model card (Mistral AI, August 2026). F1 (%).
| Rank | Model | Score | Source | Date |
|---|---|---|---|---|
| 1 | GPT-OSS-Safeguard-20B | 94.9% | Mistral AI model card | 2026-08 |
| 2 | Shieldstral 1.0 | 94.6% | Mistral AI model card | 2026-08 |
| 3 | Qwen3Guard-8B | 92.6% | Mistral AI model card | 2026-08 |
| 4 | WildGuard-7B | 92.5% | Mistral AI model card | 2026-08 |
| 5 | PolyGuard-Qwen-7B | 82.1% | Mistral AI model card | 2026-08 |
Scores sourced from Mistral AI's published model card, shown for context. Not Benchgen measurements. LlamaGuard-4, ShieldGemma, and Nemotron-3.5-Content-Safety variants do not report refusal-detection scores in this comparison.
XSTest (Refusal) on Benchgen
No Benchgen results yet — be the first to run XSTest (Refusal).
XSTest (Refusal) vs Other Benchmarks
| Benchmark | What it tests | Test size | Saturation |
|---|---|---|---|
| XSTest (Refusal) | Over/under-refusal detection | 450 | High |
| WildGuardTest (Refusal) | Refusal vs. compliance detection | 1,725 | Medium |
| PolyGuard (Refusal) | Multilingual refusal detection | 29,325 | Medium |
Run XSTest (Refusal) on Your Model
Benchgen tracks version-controlled, regression-tested guard-model performance — submit your classifier's XSTest refusal scores to compare against Shieldstral and other guard models.
Related Models & Benchmarks
- Models: Shieldstral 1.0 · GPT-OSS-Safeguard-20B
- Benchmarks: XSTest (Harm) · WildGuardTest (Refusal)
Frequently Asked Questions
What is XSTest (Refusal)?
It's the refusal-detection task on the XSTest suite, scoring whether a classifier can correctly tell if a model refused or complied with a request — particularly on prompts designed to trigger over-refusal.What is "over-refusal"?
Over-refusal is when a model declines to answer a genuinely safe request because it superficially resembles an unsafe one — a common exaggerated-safety failure mode that hurts usability.Who created XSTest?
Paul Röttger and colleagues, published in 2023.Last updated 2026-08-12.