BeaverTails — Results
| Rank | Model | Score |
|---|---|---|
| 1 | qwen3guard-8b | 85.9 |
| 2 | shieldstral-1-0 | 85 |
| 3 | gpt-oss-safeguard-20b | 83.8 |
| 4 | nemotron-3-5-content-safety-4b | 83.3 |
| 5 | llamaguard-4-12b | 69.8 |
| 6 | shieldgemma-9b | 54 |
BeaverTails
1 phaseActive
PKU Alignment's QA-pair safety classification dataset spanning 14 harm categories. Ji et al., 2023.
Overview
BeaverTails
Quick answer: BeaverTails measures QA-pair harm classification across 14 harm categories — including animal abuse, child abuse, financial crime, hate speech, self-harm, and terrorism — using a large-scale, human-labeled dataset built by the PKU Alignment team. Each QA pair can be tagged with multiple categories.
At a Glance
What it tests: Whether a classifier correctly identifies a model's answer to a question as harmful, across 14 distinct harm categories that can co-occur on a single example. Why it matters: BeaverTails is one of the largest human-labeled safety datasets (300K+ QA pairs total), giving guard-model developers a broad, category-rich training and evaluation resource beyond the narrower category sets used by earlier benchmarks. Known limitations: English-only; the full dataset is large (300K+ pairs), so guard-model papers typically evaluate on a held-out evaluation subset rather than the full release — subset composition can vary between papers.
What BeaverTails Measures
BeaverTails pairs questions with model-generated answers, each human-labeled across 14 harm categories: animal abuse, child abuse, controversial topics/politics, discrimination/stereotype/injustice, drug abuse/weapons/banned substances, financial/property crime, hate speech/offensive language, misinformation, non-violent unethical behavior, privacy violation, self-harm, sexually explicit content, terrorism/organized crime, and violence/incitement. A single QA pair can be flagged under multiple categories simultaneously, reflecting how real harmful content often spans more than one taxonomy bucket.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Safety / content moderation |
| Metric | F1 score (%) |
| Harm categories | 14 (multi-label) |
| Created by | Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, Yaodong Yang (PKU Alignment) |
| Paper | BeaverTails (NeurIPS 2023) |
| GitHub | PKU-Alignment/beavertails |
| Dataset | PKU-Alignment/BeaverTails |
| License | CC-BY-NC-4.0 |
How BeaverTails Is Scored
Each evaluated QA pair's ground-truth harm label(s) are compared against a classifier's prediction and scored as F1.
State-of-the-Art Results
Scores from the Shieldstral model card (Mistral AI, August 2026). F1 (%).
| Rank | Model | Score | Source | Date |
|---|---|---|---|---|
| 1 | Qwen3Guard-8B | 85.9% | Mistral AI model card | 2026-08 |
| 2 | Shieldstral 1.0 | 85% | Mistral AI model card | 2026-08 |
| 3 | GPT-OSS-Safeguard-20B | 83.8% | Mistral AI model card | 2026-08 |
| 4 | Nemotron-3.5-Content-Safety-4B | 83.3% | Mistral AI model card | 2026-08 |
| 5 | LlamaGuard-4-12B | 69.8% | Mistral AI model card | 2026-08 |
| 6 | ShieldGemma-9B | 54% | Mistral AI model card | 2026-08 |
Scores sourced from Mistral AI's published model card, shown for context. Not Benchgen measurements.
BeaverTails on Benchgen
No Benchgen results yet — be the first to run BeaverTails.
BeaverTails vs Other Benchmarks
| Benchmark | What it tests | Categories | Saturation |
|---|---|---|---|
| BeaverTails | Multi-label QA-pair harm classification | 14 | Medium |
| Aegis v2 (Response) | Fine-grained taxonomy response classification | 12+9 | Medium |
| HarmBench (Response) | Jailbreak success detection | N/A (behavior-based) | Medium |
Run BeaverTails on Your Model
Benchgen tracks version-controlled, regression-tested guard-model performance — submit your classifier's BeaverTails scores to compare against Shieldstral and other guard models.
Related Models & Benchmarks
- Models: Shieldstral 1.0 · GPT-OSS-Safeguard-20B
- Benchmarks: Aegis v2 (Response) · HarmBench (Response)
Frequently Asked Questions
What is BeaverTails?
BeaverTails is a large-scale, human-labeled QA-pair safety dataset spanning 14 harm categories, released by the PKU Alignment team.Can one example belong to multiple harm categories?
Yes — BeaverTails uses multi-label annotation, so a single QA pair can be tagged with more than one of the 14 categories.Who created BeaverTails?
Researchers from Peking University's PKU Alignment team (Ji et al., NeurIPS 2023).Last updated 2026-08-12.