# PolyGuard Prompt ![Category](https://img.shields.io/badge/Category-Multilingual%20Safety-1f6feb) ![Metric](https://img.shields.io/badge/Metric-F1%20Score-2da44e) ![Saturation](https://img.shields.io/badge/Saturation-Medium-d29922) [![Paper](https://img.shields.io/badge/Paper-arXiv%202504.04377-b31b1b)](https://arxiv.org/abs/2504.04377) > **Quick answer:** PolyGuard Prompt measures prompt-harm classification across 17 languages using PolyGuardPrompts, a 29,325-item evaluation benchmark combining naturally occurring multilingual human-LLM interactions with human-verified machine translations of WildGuardMix. It's one of the largest multilingual safety test sets available. ## At a Glance **What it tests:** Whether a classifier correctly flags harmful prompts across 17 languages including Arabic, Chinese, Czech, German, Hindi, Japanese, Korean, Russian, Spanish, and Thai, among others. **Why it matters:** Most safety classifiers are trained and evaluated primarily in English; PolyGuard was built to close that gap, and its authors report that PolyGuard-trained models outperform existing open-weight and commercial safety classifiers by 5.5% on average across languages. **Known limitations:** Language coverage (17 languages) is broad but not exhaustive; translation-derived items may carry some translation-artifact noise even with human verification. ## What PolyGuard Prompt Measures PolyGuardPrompts contains 29,325 items across 17 languages, built by combining real multilingual human–LLM interactions with human-verified machine translations of the English-only WildGuardMix dataset. This task scores the `prompt_harm_label` field — testing whether a classifier generalizes its harm judgments beyond English to a genuinely multilingual distribution of prompts. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Safety / multilingual content moderation | | Metric | F1 score (%) | | Test set size | 29,325 items | | Languages | 17 | | Created by | Priyanshu Kumar, Devansh Jain, Akhila Yerukola, Liwei Jiang, Himanshu Beniwal, Thomas Hartvigsen, Maarten Sap | | Paper | [PolyGuard (arXiv 2504.04377)](https://arxiv.org/abs/2504.04377) | | Dataset | [ToxicityPrompts/PolyGuardPrompts](https://huggingface.co/datasets/ToxicityPrompts/PolyGuardPrompts) | | License | CC-BY-4.0 | ## How PolyGuard Prompt Is Scored Each of the 29,325 items' ground-truth prompt-harm label is compared against a classifier's prediction and scored as F1, aggregated across all 17 languages. ## State-of-the-Art Results Scores from the Shieldstral model card (Mistral AI, August 2026). F1 (%). | Rank | Model | Score | Source | Date | |---|---|---|---|---| | 1 | [Shieldstral 1.0](/models/mistral-ai/shieldstral-1-0) | **84.6%** | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | | 2 | [Qwen3Guard-8B](/models/alibaba/qwen3guard-8b) | 84.3% | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | | 3 | [GPT-OSS-Safeguard-20B](/models/openai/gpt-oss-safeguard-20b) | 83% | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | | 4 | [Nemotron-3.5-Content-Safety-4B](/models/nvidia/nemotron-3-5-content-safety-4b) | 80.5% | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | | 5 | [LlamaGuard-4-12B](/models/meta/llamaguard-4-12b) | 62.1% | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | | 6 | [ShieldGemma-9B](/models/google/shieldgemma-9b) | 33.8% | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | *Scores sourced from Mistral AI's published model card, shown for context. Not Benchgen measurements.* ## PolyGuard Prompt on Benchgen > No Benchgen results yet — [be the first to run PolyGuard Prompt](/benchmarks/toxicityprompts/polyguard-prompt). ## PolyGuard Prompt vs Other Benchmarks | Benchmark | What it tests | Languages | Saturation | |---|---|---|---| | **PolyGuard Prompt** | Multilingual prompt-harm classification | 17 | Medium | | RTP-LX Prompt | Multilingual toxicity classification | 28 | Medium | | WildGuardTest (Prompt) | English-only prompt-harm classification | 1 | Medium | ## Run PolyGuard Prompt on Your Model Benchgen tracks version-controlled, regression-tested guard-model performance — submit your classifier's PolyGuard scores to compare against Shieldstral and other guard models. ## Related Models & Benchmarks - Models: [Shieldstral 1.0](/models/mistral-ai/shieldstral-1-0) · [PolyGuard-Qwen-7B](/models/toxicityprompts/polyguard-qwen-7b) - Benchmarks: [PolyGuard Response](/benchmarks/toxicityprompts/polyguard-response) · [RTP-LX Prompt](/benchmarks/microsoft/rtp-lx-prompt) ## Frequently Asked Questions

What is PolyGuard?

PolyGuard is a multilingual safety moderation model and dataset family covering 17 languages. PolyGuardPrompts, the evaluation set used here, contains 29,325 human-verified prompts.

Which languages does PolyGuard cover?

Arabic, Chinese, Czech, English, German, Hindi, Italian, Japanese, Korean, Dutch, Polish, Portuguese, Russian, Swedish, Thai, and others — 17 in total.

Who created PolyGuard?

Priyanshu Kumar, Devansh Jain, Akhila Yerukola, and colleagues (Kumar et al., 2025).

--- *Last updated 2026-08-12.*

Rank	Model	Score
1	shieldstral-1-0	84.6
2	qwen3guard-8b	84.3
3	gpt-oss-safeguard-20b	83
4	nemotron-3-5-content-safety-4b	80.5
5	llamaguard-4-12b	62.1
6	shieldgemma-9b	33.8

PolyGuard Prompt — Results

PolyGuard Prompt — Phases

PolyGuard Prompt