PolyGuard Prompt — Results
| Rank | Model | Score |
|---|---|---|
| 1 | shieldstral-1-0 | 84.6 |
| 2 | qwen3guard-8b | 84.3 |
| 3 | gpt-oss-safeguard-20b | 83 |
| 4 | nemotron-3-5-content-safety-4b | 80.5 |
| 5 | llamaguard-4-12b | 62.1 |
| 6 | shieldgemma-9b | 33.8 |
PolyGuard Prompt
1 phaseActive
A 29,325-item multilingual safety benchmark spanning 17 languages, prompt-classification task. Kumar et al., 2025.
Overview
PolyGuard Prompt
Quick answer: PolyGuard Prompt measures prompt-harm classification across 17 languages using PolyGuardPrompts, a 29,325-item evaluation benchmark combining naturally occurring multilingual human-LLM interactions with human-verified machine translations of WildGuardMix. It's one of the largest multilingual safety test sets available.
At a Glance
What it tests: Whether a classifier correctly flags harmful prompts across 17 languages including Arabic, Chinese, Czech, German, Hindi, Japanese, Korean, Russian, Spanish, and Thai, among others. Why it matters: Most safety classifiers are trained and evaluated primarily in English; PolyGuard was built to close that gap, and its authors report that PolyGuard-trained models outperform existing open-weight and commercial safety classifiers by 5.5% on average across languages. Known limitations: Language coverage (17 languages) is broad but not exhaustive; translation-derived items may carry some translation-artifact noise even with human verification.
What PolyGuard Prompt Measures
PolyGuardPrompts contains 29,325 items across 17 languages, built by combining real multilingual human–LLM interactions with human-verified machine translations of the English-only WildGuardMix dataset. This task scores the prompt_harm_label field — testing whether a classifier generalizes its harm judgments beyond English to a genuinely multilingual distribution of prompts.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Safety / multilingual content moderation |
| Metric | F1 score (%) |
| Test set size | 29,325 items |
| Languages | 17 |
| Created by | Priyanshu Kumar, Devansh Jain, Akhila Yerukola, Liwei Jiang, Himanshu Beniwal, Thomas Hartvigsen, Maarten Sap |
| Paper | PolyGuard (arXiv 2504.04377) |
| Dataset | ToxicityPrompts/PolyGuardPrompts |
| License | CC-BY-4.0 |
How PolyGuard Prompt Is Scored
Each of the 29,325 items' ground-truth prompt-harm label is compared against a classifier's prediction and scored as F1, aggregated across all 17 languages.
State-of-the-Art Results
Scores from the Shieldstral model card (Mistral AI, August 2026). F1 (%).
| Rank | Model | Score | Source | Date |
|---|---|---|---|---|
| 1 | Shieldstral 1.0 | 84.6% | Mistral AI model card | 2026-08 |
| 2 | Qwen3Guard-8B | 84.3% | Mistral AI model card | 2026-08 |
| 3 | GPT-OSS-Safeguard-20B | 83% | Mistral AI model card | 2026-08 |
| 4 | Nemotron-3.5-Content-Safety-4B | 80.5% | Mistral AI model card | 2026-08 |
| 5 | LlamaGuard-4-12B | 62.1% | Mistral AI model card | 2026-08 |
| 6 | ShieldGemma-9B | 33.8% | Mistral AI model card | 2026-08 |
Scores sourced from Mistral AI's published model card, shown for context. Not Benchgen measurements.
PolyGuard Prompt on Benchgen
No Benchgen results yet — be the first to run PolyGuard Prompt.
PolyGuard Prompt vs Other Benchmarks
| Benchmark | What it tests | Languages | Saturation |
|---|---|---|---|
| PolyGuard Prompt | Multilingual prompt-harm classification | 17 | Medium |
| RTP-LX Prompt | Multilingual toxicity classification | 28 | Medium |
| WildGuardTest (Prompt) | English-only prompt-harm classification | 1 | Medium |
Run PolyGuard Prompt on Your Model
Benchgen tracks version-controlled, regression-tested guard-model performance — submit your classifier's PolyGuard scores to compare against Shieldstral and other guard models.
Related Models & Benchmarks
- Models: Shieldstral 1.0 · PolyGuard-Qwen-7B
- Benchmarks: PolyGuard Response · RTP-LX Prompt
Frequently Asked Questions
What is PolyGuard?
PolyGuard is a multilingual safety moderation model and dataset family covering 17 languages. PolyGuardPrompts, the evaluation set used here, contains 29,325 human-verified prompts.Which languages does PolyGuard cover?
Arabic, Chinese, Czech, English, German, Hindi, Italian, Japanese, Korean, Dutch, Polish, Portuguese, Russian, Swedish, Thai, and others — 17 in total.Who created PolyGuard?
Priyanshu Kumar, Devansh Jain, Akhila Yerukola, and colleagues (Kumar et al., 2025).Last updated 2026-08-12.