# XSTest (Refusal) ![Category](https://img.shields.io/badge/Category-Safety-1f6feb) ![Metric](https://img.shields.io/badge/Metric-F1%20Score-2da44e) ![Saturation](https://img.shields.io/badge/Saturation-High-d29922) [![Paper](https://img.shields.io/badge/Paper-arXiv%202308.01263-b31b1b)](https://arxiv.org/abs/2308.01263) [![GitHub](https://img.shields.io/badge/GitHub-paul--rottger%2Fexaggerated--safety-181717?logo=github&logoColor=white)](https://github.com/paul-rottger/exaggerated-safety) > **Quick answer:** XSTest (Refusal) measures whether a classifier can correctly detect over-refusal — a target model declining to answer a genuinely safe prompt because it superficially resembles an unsafe one. It uses the same 450-item XSTest suite as the harm-classification task but scores refusal detection on model completions specifically. ## At a Glance **What it tests:** Whether a classifier can distinguish a genuine refusal from a compliant answer, applied to completions generated on XSTest's 250 safe / 200 unsafe prompt set. **Why it matters:** Over-refusal is a real UX cost in production systems — models that decline safe requests because of surface-level similarity to unsafe ones frustrate users. A refusal classifier needs to catch this accurately to support over-refusal monitoring. **Known limitations:** Small test set (450 items); results depend on which target model's completions were used to generate the refusal/compliance labels being evaluated. ## What XSTest (Refusal) Measures This task scores whether a classifier correctly labels a model's response to an XSTest prompt as a refusal or a compliance. Applied across both the safe and unsafe halves of the set, it surfaces two distinct failure modes: refusing a safe prompt (over-refusal) and complying with an unsafe prompt (under-refusal) — with the safe-prompt half being the more diagnostic test for exaggerated safety specifically. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Safety / refusal detection | | Metric | F1 score (%) | | Test set size | 450 (250 safe + 200 unsafe) | | Created by | Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, Dirk Hovy | | Paper | [XSTest (arXiv 2308.01263)](https://arxiv.org/abs/2308.01263) | | GitHub | [paul-rottger/exaggerated-safety](https://github.com/paul-rottger/exaggerated-safety) | | Dataset | [walledai/XSTest](https://huggingface.co/datasets/walledai/XSTest) | | License | CC-BY-4.0 (prompts) | ## How XSTest (Refusal) Is Scored Each response's ground-truth refusal/compliance label is compared against a classifier's prediction and scored as F1 across the 450-item set. ## State-of-the-Art Results Scores from the Shieldstral model card (Mistral AI, August 2026). F1 (%). | Rank | Model | Score | Source | Date | |---|---|---|---|---| | 1 | [GPT-OSS-Safeguard-20B](/models/openai/gpt-oss-safeguard-20b) | **94.9%** | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | | 2 | [Shieldstral 1.0](/models/mistral-ai/shieldstral-1-0) | 94.6% | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | | 3 | [Qwen3Guard-8B](/models/alibaba/qwen3guard-8b) | 92.6% | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | | 4 | [WildGuard-7B](/models/allen-institute-for-ai/wildguard-7b) | 92.5% | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | | 5 | [PolyGuard-Qwen-7B](/models/toxicityprompts/polyguard-qwen-7b) | 82.1% | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | *Scores sourced from Mistral AI's published model card, shown for context. Not Benchgen measurements. LlamaGuard-4, ShieldGemma, and Nemotron-3.5-Content-Safety variants do not report refusal-detection scores in this comparison.* ## XSTest (Refusal) on Benchgen > No Benchgen results yet — [be the first to run XSTest (Refusal)](/benchmarks/independent-research/xstest-refusal). ## XSTest (Refusal) vs Other Benchmarks | Benchmark | What it tests | Test size | Saturation | |---|---|---|---| | **XSTest (Refusal)** | Over/under-refusal detection | 450 | High | | WildGuardTest (Refusal) | Refusal vs. compliance detection | 1,725 | Medium | | PolyGuard (Refusal) | Multilingual refusal detection | 29,325 | Medium | ## Run XSTest (Refusal) on Your Model Benchgen tracks version-controlled, regression-tested guard-model performance — submit your classifier's XSTest refusal scores to compare against Shieldstral and other guard models. ## Related Models & Benchmarks - Models: [Shieldstral 1.0](/models/mistral-ai/shieldstral-1-0) · [GPT-OSS-Safeguard-20B](/models/openai/gpt-oss-safeguard-20b) - Benchmarks: [XSTest (Harm)](/benchmarks/independent-research/xstest-harm) · [WildGuardTest (Refusal)](/benchmarks/allen-institute-for-ai/wildguardtest-refusal) ## Frequently Asked Questions

What is XSTest (Refusal)?

It's the refusal-detection task on the XSTest suite, scoring whether a classifier can correctly tell if a model refused or complied with a request — particularly on prompts designed to trigger over-refusal.

What is "over-refusal"?

Over-refusal is when a model declines to answer a genuinely safe request because it superficially resembles an unsafe one — a common exaggerated-safety failure mode that hurts usability.

Who created XSTest?

Paul Röttger and colleagues, published in 2023.

--- *Last updated 2026-08-12.*

Rank	Model	Score
1	gpt-oss-safeguard-20b	94.9
2	shieldstral-1-0	94.6
3	qwen3guard-8b	92.6
4	wildguard-7b	92.5
5	polyguard-qwen-7b	82.1

XSTest (Refusal) — Results

XSTest (Refusal) — Evaluations