ToxicChat — Results
| Rank | Model | Score |
|---|---|---|
| 1 | shieldstral-1-0 | 84.1 |
| 2 | gpt-oss-safeguard-20b | 79.8 |
| 3 | qwen3guard-8b | 75.6 |
| 4 | nemotron-3-5-content-safety-4b | 72.2 |
| 5 | shieldgemma-9b | 62.4 |
| 6 | llamaguard-4-12b | 51 |
ToxicChat
1 phaseActive
LMSYS's real-world toxicity dataset sourced from the Vicuna online demo. 10,165 human-annotated user prompts, F1 metric. From Lin et al., 2023.
Overview
ToxicChat
Quick answer: ToxicChat measures toxicity detection on real user prompts collected from LMSYS's Vicuna online demo — not synthetic red-team data. It contains 10,165 human-annotated prompts and is notable for including in-the-wild jailbreak attempts, making it a harder, more realistic test than curated red-teaming datasets.
At a Glance
What it tests: Whether a classifier can detect toxic content in real, unfiltered user-to-chatbot queries, including naturally occurring jailbreak attempts. Why it matters: Because ToxicChat is sourced from genuine demo traffic rather than crowdsourced red-teaming, it exposes distribution shifts that curated benchmarks miss — the original paper showed existing toxicity APIs (including OpenAI's Moderation API) underperform on it relative to curated datasets. Known limitations: English-only; sourced from a single chatbot demo (Vicuna), so may not represent toxicity patterns across other deployment contexts.
What ToxicChat Measures
ToxicChat annotates 10,165 user prompts drawn from the Vicuna online demo for toxicity and jailbreaking intent, using a human-AI collaborative annotation framework. Roughly 7.2% of prompts are labeled toxic and 1.8% are labeled jailbreaking attempts — a realistic, imbalanced distribution rather than a curated 50/50 split. The dataset is versioned (e.g., toxicchat0124), with each release incorporating annotation error corrections.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Safety / toxicity detection |
| Metric | F1 score (%) |
| Test set size | 10,165 user prompts |
| Toxic rate | ~7.18% |
| Created by | Zi Lin, Zihan Wang, Yongqi Tong, Yangkun Wang, Yuxin Guo, Yujia Wang, Jingbo Shang (LMSYS / UCSD) |
| Paper | ToxicChat (arXiv 2310.17389) |
| Dataset | lmsys/toxic-chat |
| License | CC-BY-NC-4.0 |
How ToxicChat Is Scored
Each prompt's ground-truth toxicity label is compared to the classifier's prediction across the test set, scored as F1.
State-of-the-Art Results
Scores from the Shieldstral model card (Mistral AI, August 2026). F1 (%).
| Rank | Model | Score | Source | Date |
|---|---|---|---|---|
| 1 | Shieldstral 1.0 | 84.1% | Mistral AI model card | 2026-08 |
| 2 | GPT-OSS-Safeguard-20B | 79.8% | Mistral AI model card | 2026-08 |
| 3 | Qwen3Guard-8B | 75.6% | Mistral AI model card | 2026-08 |
| 4 | Nemotron-3.5-Content-Safety-4B | 72.2% | Mistral AI model card | 2026-08 |
| 5 | ShieldGemma-9B | 62.4% | Mistral AI model card | 2026-08 |
| 6 | LlamaGuard-4-12B | 51.0% | Mistral AI model card | 2026-08 |
Scores sourced from Mistral AI's published model card, shown for context. Not Benchgen measurements.
ToxicChat on Benchgen
No Benchgen results yet — be the first to run ToxicChat.
ToxicChat vs Other Benchmarks
| Benchmark | What it tests | Test size | Saturation |
|---|---|---|---|
| ToxicChat | Real-world chatbot query toxicity | 10,165 | Medium |
| WildGuardTest (Prompt) | Mixed synthetic + in-the-wild prompt harm | 1,725 | Medium |
| OpenAI Moderation | Multi-category content moderation | 1,680 | Medium |
Run ToxicChat on Your Model
Benchgen tracks version-controlled, regression-tested guard-model performance — submit your classifier's ToxicChat scores to compare against Shieldstral and other guard models.
Related Models & Benchmarks
- Models: Shieldstral 1.0 · GPT-OSS-Safeguard-20B
- Benchmarks: WildGuardTest (Prompt) · OpenAI Moderation
Frequently Asked Questions
What is ToxicChat?
ToxicChat is a toxicity-detection benchmark built from 10,165 real user prompts collected from LMSYS's Vicuna online demo, human-annotated for toxicity and jailbreaking intent.How is ToxicChat different from synthetic safety benchmarks?
It's sourced from genuine chatbot traffic rather than curated red-team prompts, so it captures natural distribution shifts and in-the-wild jailbreak attempts that synthetic datasets can miss.Who created ToxicChat?
Researchers at LMSYS and UC San Diego (Lin et al., 2023).Last updated 2026-08-12.