# ToxicChat ![Category](https://img.shields.io/badge/Category-Safety-1f6feb) ![Metric](https://img.shields.io/badge/Metric-F1%20Score-2da44e) ![Saturation](https://img.shields.io/badge/Saturation-Medium-d29922) [![Paper](https://img.shields.io/badge/Paper-arXiv%202310.17389-b31b1b)](https://arxiv.org/abs/2310.17389) > **Quick answer:** ToxicChat measures toxicity detection on real user prompts collected from LMSYS's Vicuna online demo — not synthetic red-team data. It contains 10,165 human-annotated prompts and is notable for including in-the-wild jailbreak attempts, making it a harder, more realistic test than curated red-teaming datasets. ## At a Glance **What it tests:** Whether a classifier can detect toxic content in real, unfiltered user-to-chatbot queries, including naturally occurring jailbreak attempts. **Why it matters:** Because ToxicChat is sourced from genuine demo traffic rather than crowdsourced red-teaming, it exposes distribution shifts that curated benchmarks miss — the original paper showed existing toxicity APIs (including OpenAI's Moderation API) underperform on it relative to curated datasets. **Known limitations:** English-only; sourced from a single chatbot demo (Vicuna), so may not represent toxicity patterns across other deployment contexts. ## What ToxicChat Measures ToxicChat annotates 10,165 user prompts drawn from the Vicuna online demo for toxicity and jailbreaking intent, using a human-AI collaborative annotation framework. Roughly 7.2% of prompts are labeled toxic and 1.8% are labeled jailbreaking attempts — a realistic, imbalanced distribution rather than a curated 50/50 split. The dataset is versioned (e.g., `toxicchat0124`), with each release incorporating annotation error corrections. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Safety / toxicity detection | | Metric | F1 score (%) | | Test set size | 10,165 user prompts | | Toxic rate | ~7.18% | | Created by | Zi Lin, Zihan Wang, Yongqi Tong, Yangkun Wang, Yuxin Guo, Yujia Wang, Jingbo Shang (LMSYS / UCSD) | | Paper | [ToxicChat (arXiv 2310.17389)](https://arxiv.org/abs/2310.17389) | | Dataset | [lmsys/toxic-chat](https://huggingface.co/datasets/lmsys/toxic-chat) | | License | CC-BY-NC-4.0 | ## How ToxicChat Is Scored Each prompt's ground-truth toxicity label is compared to the classifier's prediction across the test set, scored as F1. ## State-of-the-Art Results Scores from the Shieldstral model card (Mistral AI, August 2026). F1 (%). | Rank | Model | Score | Source | Date | |---|---|---|---|---| | 1 | [Shieldstral 1.0](/models/mistral-ai/shieldstral-1-0) | **84.1%** | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | | 2 | [GPT-OSS-Safeguard-20B](/models/openai/gpt-oss-safeguard-20b) | 79.8% | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | | 3 | [Qwen3Guard-8B](/models/alibaba/qwen3guard-8b) | 75.6% | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | | 4 | [Nemotron-3.5-Content-Safety-4B](/models/nvidia/nemotron-3-5-content-safety-4b) | 72.2% | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | | 5 | [ShieldGemma-9B](/models/google/shieldgemma-9b) | 62.4% | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | | 6 | [LlamaGuard-4-12B](/models/meta/llamaguard-4-12b) | 51.0% | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | *Scores sourced from Mistral AI's published model card, shown for context. Not Benchgen measurements.* ## ToxicChat on Benchgen > No Benchgen results yet — [be the first to run ToxicChat](/benchmarks/lmsys/toxicchat). ## ToxicChat vs Other Benchmarks | Benchmark | What it tests | Test size | Saturation | |---|---|---|---| | **ToxicChat** | Real-world chatbot query toxicity | 10,165 | Medium | | WildGuardTest (Prompt) | Mixed synthetic + in-the-wild prompt harm | 1,725 | Medium | | OpenAI Moderation | Multi-category content moderation | 1,680 | Medium | ## Run ToxicChat on Your Model Benchgen tracks version-controlled, regression-tested guard-model performance — submit your classifier's ToxicChat scores to compare against Shieldstral and other guard models. ## Related Models & Benchmarks - Models: [Shieldstral 1.0](/models/mistral-ai/shieldstral-1-0) · [GPT-OSS-Safeguard-20B](/models/openai/gpt-oss-safeguard-20b) - Benchmarks: [WildGuardTest (Prompt)](/benchmarks/allen-institute-for-ai/wildguardtest-prompt) · [OpenAI Moderation](/benchmarks/openai/openai-moderation) ## Frequently Asked Questions

What is ToxicChat?

ToxicChat is a toxicity-detection benchmark built from 10,165 real user prompts collected from LMSYS's Vicuna online demo, human-annotated for toxicity and jailbreaking intent.

How is ToxicChat different from synthetic safety benchmarks?

It's sourced from genuine chatbot traffic rather than curated red-team prompts, so it captures natural distribution shifts and in-the-wild jailbreak attempts that synthetic datasets can miss.

Who created ToxicChat?

Researchers at LMSYS and UC San Diego (Lin et al., 2023).

--- *Last updated 2026-08-12.*

Rank	Model	Score
1	shieldstral-1-0	84.1
2	gpt-oss-safeguard-20b	79.8
3	qwen3guard-8b	75.6
4	nemotron-3-5-content-safety-4b	72.2
5	shieldgemma-9b	62.4
6	llamaguard-4-12b	51

ToxicChat — Results

ToxicChat — Evaluations