# PolyGuard (Refusal) ![Category](https://img.shields.io/badge/Category-Multilingual%20Safety-1f6feb) ![Metric](https://img.shields.io/badge/Metric-F1%20Score-2da44e) ![Saturation](https://img.shields.io/badge/Saturation-Medium-d29922) [![Paper](https://img.shields.io/badge/Paper-arXiv%202504.04377-b31b1b)](https://arxiv.org/abs/2504.04377) > **Quick answer:** PolyGuard (Refusal) measures whether a classifier can detect refusal vs. compliance across 17 languages, using the same 29,325-item PolyGuardPrompts set as the prompt and response tasks, but scoring the `response_refusal_label` field. ## At a Glance **What it tests:** Whether a classifier correctly labels a model's multilingual response as a refusal or a compliance, independent of harm. **Why it matters:** Refusal detection at scale across 17 languages helps teams monitor over-refusal and under-refusal patterns in multilingual deployments, where refusal-triggering phrases and cultural norms around sensitive topics vary by language. **Known limitations:** Refusal phrasing conventions differ across languages and models, which can introduce more label ambiguity than in English-only refusal benchmarks like XSTest. ## What PolyGuard (Refusal) Measures This task scores the `response_refusal_label` field on the 29,325-item PolyGuardPrompts set — testing whether a classifier's refusal/compliance judgment generalizes across the same 17-language distribution used for prompt- and response-harm classification. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Safety / multilingual refusal detection | | Metric | F1 score (%) | | Test set size | 29,325 items | | Languages | 17 | | Created by | Priyanshu Kumar, Devansh Jain, Akhila Yerukola, Liwei Jiang, Himanshu Beniwal, Thomas Hartvigsen, Maarten Sap | | Paper | [PolyGuard (arXiv 2504.04377)](https://arxiv.org/abs/2504.04377) | | Dataset | [ToxicityPrompts/PolyGuardPrompts](https://huggingface.co/datasets/ToxicityPrompts/PolyGuardPrompts) | | License | CC-BY-4.0 | ## How PolyGuard (Refusal) Is Scored Each item's ground-truth refusal label is compared against a classifier's prediction and scored as F1, aggregated across all 17 languages. ## State-of-the-Art Results Scores from the Shieldstral model card (Mistral AI, August 2026). F1 (%). | Rank | Model | Score | Source | Date | |---|---|---|---|---| | 1 | [GPT-OSS-Safeguard-20B](/models/openai/gpt-oss-safeguard-20b) | **92.3%** | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | | 2 | [WildGuard-7B](/models/allen-institute-for-ai/wildguard-7b) | 89.6% | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | | 3 | [Shieldstral 1.0](/models/mistral-ai/shieldstral-1-0) | 89.5% | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | | 4 | [Qwen3Guard-8B](/models/alibaba/qwen3guard-8b) | 89.3% | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | | 5 | [PolyGuard-Qwen-7B](/models/toxicityprompts/polyguard-qwen-7b) | 83.8% | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | *Scores sourced from Mistral AI's published model card, shown for context. Not Benchgen measurements. LlamaGuard-4, ShieldGemma, and Nemotron-3.5-Content-Safety variants do not report refusal-detection scores in this comparison.* ## PolyGuard (Refusal) on Benchgen > No Benchgen results yet — [be the first to run PolyGuard (Refusal)](/benchmarks/toxicityprompts/polyguard-refusal). ## PolyGuard (Refusal) vs Other Benchmarks | Benchmark | What it tests | Languages | Saturation | |---|---|---|---| | **PolyGuard (Refusal)** | Multilingual refusal detection | 17 | Medium | | WildGuardTest (Refusal) | English-only refusal detection | 1 | Medium | | XSTest (Refusal) | Over-refusal detection | 1 | High | ## Run PolyGuard (Refusal) on Your Model Benchgen tracks version-controlled, regression-tested guard-model performance — submit your classifier's PolyGuard refusal scores to compare against Shieldstral and other guard models. ## Related Models & Benchmarks - Models: [Shieldstral 1.0](/models/mistral-ai/shieldstral-1-0) · [GPT-OSS-Safeguard-20B](/models/openai/gpt-oss-safeguard-20b) - Benchmarks: [PolyGuard Prompt](/benchmarks/toxicityprompts/polyguard-prompt) · [PolyGuard Response](/benchmarks/toxicityprompts/polyguard-response) ## Frequently Asked Questions

What is PolyGuard (Refusal)?

It's the refusal-detection task on the 29,325-item, 17-language PolyGuardPrompts benchmark.

How does this differ from XSTest (Refusal)?

XSTest is English-only and focused on exaggerated-safety trap prompts; PolyGuard (Refusal) tests general refusal detection across 17 languages on naturally occurring and translated data.

Who created PolyGuard?

Priyanshu Kumar, Devansh Jain, Akhila Yerukola, and colleagues (Kumar et al., 2025).

--- *Last updated 2026-08-12.*

Rank	Model	Score
1	gpt-oss-safeguard-20b	92.3
2	wildguard-7b	89.6
3	shieldstral-1-0	89.5
4	qwen3guard-8b	89.3
5	polyguard-qwen-7b	83.8

PolyGuard (Refusal) — Results

PolyGuard (Refusal) — Phases

PolyGuard (Refusal)