# PolyGuard Response ![Category](https://img.shields.io/badge/Category-Multilingual%20Safety-1f6feb) ![Metric](https://img.shields.io/badge/Metric-F1%20Score-2da44e) ![Saturation](https://img.shields.io/badge/Saturation-Medium-d29922) [![Paper](https://img.shields.io/badge/Paper-arXiv%202504.04377-b31b1b)](https://arxiv.org/abs/2504.04377) > **Quick answer:** PolyGuard Response measures multilingual response-harm classification across 17 languages, using the same 29,325-item PolyGuardPrompts set as PolyGuard Prompt but scoring the model's reply rather than the user's request. ## At a Glance **What it tests:** Whether a classifier correctly flags harmful model responses across 17 languages, given the prompt-response pair. **Why it matters:** Response-harm judgments must account for both the response text and its language-specific cultural/linguistic context — a harder generalization test than prompt-only classification, especially across low-resource languages. **Known limitations:** Some languages in the 17-language set have less training/eval data than others, so per-language reliability may vary even though the aggregate F1 is reported as a single number. ## What PolyGuard Response Measures This task scores the `response_harm_label` field on the 29,325-item PolyGuardPrompts set, testing whether a classifier's judgment of response harmfulness generalizes across the same 17 languages used for prompt classification. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Safety / multilingual content moderation | | Metric | F1 score (%) | | Test set size | 29,325 items | | Languages | 17 | | Created by | Priyanshu Kumar, Devansh Jain, Akhila Yerukola, Liwei Jiang, Himanshu Beniwal, Thomas Hartvigsen, Maarten Sap | | Paper | [PolyGuard (arXiv 2504.04377)](https://arxiv.org/abs/2504.04377) | | Dataset | [ToxicityPrompts/PolyGuardPrompts](https://huggingface.co/datasets/ToxicityPrompts/PolyGuardPrompts) | | License | CC-BY-4.0 | ## How PolyGuard Response Is Scored Each item's ground-truth response-harm label is compared against a classifier's prediction and scored as F1, aggregated across all 17 languages. ## State-of-the-Art Results Scores from the Shieldstral model card (Mistral AI, August 2026). F1 (%). | Rank | Model | Score | Source | Date | |---|---|---|---|---| | 1 | [GPT-OSS-Safeguard-20B](/models/openai/gpt-oss-safeguard-20b) | **80%** | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | | 2 | [Shieldstral 1.0](/models/mistral-ai/shieldstral-1-0) | 78.3% | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | | 3 | [Qwen3Guard-8B](/models/alibaba/qwen3guard-8b) | 78.1% | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | | 4 | [Nemotron-3.5-Content-Safety-4B](/models/nvidia/nemotron-3-5-content-safety-4b) | 75.3% | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | | 5 | [LlamaGuard-4-12B](/models/meta/llamaguard-4-12b) | 54.6% | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | | 6 | [ShieldGemma-9B](/models/google/shieldgemma-9b) | 31.8% | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | *Scores sourced from Mistral AI's published model card, shown for context. Not Benchgen measurements.* ## PolyGuard Response on Benchgen > No Benchgen results yet — [be the first to run PolyGuard Response](/benchmarks/toxicityprompts/polyguard-response). ## PolyGuard Response vs Other Benchmarks | Benchmark | What it tests | Languages | Saturation | |---|---|---|---| | **PolyGuard Response** | Multilingual response-harm classification | 17 | Medium | | RTP-LX Completion | Multilingual completion toxicity | 28 | Medium | | WildGuardTest (Response) | English-only response-harm classification | 1 | Medium | ## Run PolyGuard Response on Your Model Benchgen tracks version-controlled, regression-tested guard-model performance — submit your classifier's PolyGuard response scores to compare against Shieldstral and other guard models. ## Related Models & Benchmarks - Models: [Shieldstral 1.0](/models/mistral-ai/shieldstral-1-0) · [PolyGuard-Qwen-7B](/models/toxicityprompts/polyguard-qwen-7b) - Benchmarks: [PolyGuard Prompt](/benchmarks/toxicityprompts/polyguard-prompt) · [RTP-LX Completion](/benchmarks/microsoft/rtp-lx-completion) ## Frequently Asked Questions

What is PolyGuard Response?

It's the response-harm classification task on the 29,325-item, 17-language PolyGuardPrompts benchmark.

Why does PolyGuard-Qwen-7B lead this benchmark?

PolyGuard-Qwen-7B is fine-tuned directly on PolyGuard training data, giving it a natural advantage on this specific evaluation relative to models not trained on it.

Who created PolyGuard?

Priyanshu Kumar, Devansh Jain, Akhila Yerukola, and colleagues (Kumar et al., 2025).

--- *Last updated 2026-08-12.*

Rank	Model	Score
1	gpt-oss-safeguard-20b	80
2	shieldstral-1-0	78.3
3	qwen3guard-8b	78.1
4	nemotron-3-5-content-safety-4b	75.3
5	llamaguard-4-12b	54.6
6	shieldgemma-9b	31.8

PolyGuard Response — Results

PolyGuard Response — Evaluations