PolyGuard (Refusal) — Results
| Rank | Model | Score |
|---|---|---|
| 1 | gpt-oss-safeguard-20b | 92.3 |
| 2 | wildguard-7b | 89.6 |
| 3 | shieldstral-1-0 | 89.5 |
| 4 | qwen3guard-8b | 89.3 |
| 5 | polyguard-qwen-7b | 83.8 |
PolyGuard (Refusal)
1 phaseActive
A 29,325-item multilingual safety benchmark spanning 17 languages, refusal-detection task. Kumar et al., 2025.
Overview
PolyGuard (Refusal)
Quick answer: PolyGuard (Refusal) measures whether a classifier can detect refusal vs. compliance across 17 languages, using the same 29,325-item PolyGuardPrompts set as the prompt and response tasks, but scoring the
response_refusal_labelfield.
At a Glance
What it tests: Whether a classifier correctly labels a model's multilingual response as a refusal or a compliance, independent of harm. Why it matters: Refusal detection at scale across 17 languages helps teams monitor over-refusal and under-refusal patterns in multilingual deployments, where refusal-triggering phrases and cultural norms around sensitive topics vary by language. Known limitations: Refusal phrasing conventions differ across languages and models, which can introduce more label ambiguity than in English-only refusal benchmarks like XSTest.
What PolyGuard (Refusal) Measures
This task scores the response_refusal_label field on the 29,325-item PolyGuardPrompts set — testing whether a classifier's refusal/compliance judgment generalizes across the same 17-language distribution used for prompt- and response-harm classification.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Safety / multilingual refusal detection |
| Metric | F1 score (%) |
| Test set size | 29,325 items |
| Languages | 17 |
| Created by | Priyanshu Kumar, Devansh Jain, Akhila Yerukola, Liwei Jiang, Himanshu Beniwal, Thomas Hartvigsen, Maarten Sap |
| Paper | PolyGuard (arXiv 2504.04377) |
| Dataset | ToxicityPrompts/PolyGuardPrompts |
| License | CC-BY-4.0 |
How PolyGuard (Refusal) Is Scored
Each item's ground-truth refusal label is compared against a classifier's prediction and scored as F1, aggregated across all 17 languages.
State-of-the-Art Results
Scores from the Shieldstral model card (Mistral AI, August 2026). F1 (%).
| Rank | Model | Score | Source | Date |
|---|---|---|---|---|
| 1 | GPT-OSS-Safeguard-20B | 92.3% | Mistral AI model card | 2026-08 |
| 2 | WildGuard-7B | 89.6% | Mistral AI model card | 2026-08 |
| 3 | Shieldstral 1.0 | 89.5% | Mistral AI model card | 2026-08 |
| 4 | Qwen3Guard-8B | 89.3% | Mistral AI model card | 2026-08 |
| 5 | PolyGuard-Qwen-7B | 83.8% | Mistral AI model card | 2026-08 |
Scores sourced from Mistral AI's published model card, shown for context. Not Benchgen measurements. LlamaGuard-4, ShieldGemma, and Nemotron-3.5-Content-Safety variants do not report refusal-detection scores in this comparison.
PolyGuard (Refusal) on Benchgen
No Benchgen results yet — be the first to run PolyGuard (Refusal).
PolyGuard (Refusal) vs Other Benchmarks
| Benchmark | What it tests | Languages | Saturation |
|---|---|---|---|
| PolyGuard (Refusal) | Multilingual refusal detection | 17 | Medium |
| WildGuardTest (Refusal) | English-only refusal detection | 1 | Medium |
| XSTest (Refusal) | Over-refusal detection | 1 | High |
Run PolyGuard (Refusal) on Your Model
Benchgen tracks version-controlled, regression-tested guard-model performance — submit your classifier's PolyGuard refusal scores to compare against Shieldstral and other guard models.
Related Models & Benchmarks
- Models: Shieldstral 1.0 · GPT-OSS-Safeguard-20B
- Benchmarks: PolyGuard Prompt · PolyGuard Response
Frequently Asked Questions
What is PolyGuard (Refusal)?
It's the refusal-detection task on the 29,325-item, 17-language PolyGuardPrompts benchmark.How does this differ from XSTest (Refusal)?
XSTest is English-only and focused on exaggerated-safety trap prompts; PolyGuard (Refusal) tests general refusal detection across 17 languages on naturally occurring and translated data.Who created PolyGuard?
Priyanshu Kumar, Devansh Jain, Akhila Yerukola, and colleagues (Kumar et al., 2025).Last updated 2026-08-12.