# BeaverTails ![Category](https://img.shields.io/badge/Category-Safety-1f6feb) ![Metric](https://img.shields.io/badge/Metric-F1%20Score-2da44e) ![Saturation](https://img.shields.io/badge/Saturation-Medium-d29922) [![Paper](https://img.shields.io/badge/Paper-arXiv%202307.04657-b31b1b)](https://arxiv.org/abs/2307.04657) [![GitHub](https://img.shields.io/badge/GitHub-PKU--Alignment%2Fbeavertails-181717?logo=github&logoColor=white)](https://github.com/PKU-Alignment/beavertails) > **Quick answer:** BeaverTails measures QA-pair harm classification across 14 harm categories — including animal abuse, child abuse, financial crime, hate speech, self-harm, and terrorism — using a large-scale, human-labeled dataset built by the PKU Alignment team. Each QA pair can be tagged with multiple categories. ## At a Glance **What it tests:** Whether a classifier correctly identifies a model's answer to a question as harmful, across 14 distinct harm categories that can co-occur on a single example. **Why it matters:** BeaverTails is one of the largest human-labeled safety datasets (300K+ QA pairs total), giving guard-model developers a broad, category-rich training and evaluation resource beyond the narrower category sets used by earlier benchmarks. **Known limitations:** English-only; the full dataset is large (300K+ pairs), so guard-model papers typically evaluate on a held-out evaluation subset rather than the full release — subset composition can vary between papers. ## What BeaverTails Measures BeaverTails pairs questions with model-generated answers, each human-labeled across 14 harm categories: animal abuse, child abuse, controversial topics/politics, discrimination/stereotype/injustice, drug abuse/weapons/banned substances, financial/property crime, hate speech/offensive language, misinformation, non-violent unethical behavior, privacy violation, self-harm, sexually explicit content, terrorism/organized crime, and violence/incitement. A single QA pair can be flagged under multiple categories simultaneously, reflecting how real harmful content often spans more than one taxonomy bucket. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Safety / content moderation | | Metric | F1 score (%) | | Harm categories | 14 (multi-label) | | Created by | Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, Yaodong Yang (PKU Alignment) | | Paper | [BeaverTails (NeurIPS 2023)](https://arxiv.org/abs/2307.04657) | | GitHub | [PKU-Alignment/beavertails](https://github.com/PKU-Alignment/beavertails) | | Dataset | [PKU-Alignment/BeaverTails](https://huggingface.co/datasets/PKU-Alignment/BeaverTails) | | License | CC-BY-NC-4.0 | ## How BeaverTails Is Scored Each evaluated QA pair's ground-truth harm label(s) are compared against a classifier's prediction and scored as F1. ## State-of-the-Art Results Scores from the Shieldstral model card (Mistral AI, August 2026). F1 (%). | Rank | Model | Score | Source | Date | |---|---|---|---|---| | 1 | [Qwen3Guard-8B](/models/alibaba/qwen3guard-8b) | **85.9%** | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | | 2 | [Shieldstral 1.0](/models/mistral-ai/shieldstral-1-0) | 85% | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | | 3 | [GPT-OSS-Safeguard-20B](/models/openai/gpt-oss-safeguard-20b) | 83.8% | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | | 4 | [Nemotron-3.5-Content-Safety-4B](/models/nvidia/nemotron-3-5-content-safety-4b) | 83.3% | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | | 5 | [LlamaGuard-4-12B](/models/meta/llamaguard-4-12b) | 69.8% | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | | 6 | [ShieldGemma-9B](/models/google/shieldgemma-9b) | 54% | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | *Scores sourced from Mistral AI's published model card, shown for context. Not Benchgen measurements.* ## BeaverTails on Benchgen > No Benchgen results yet — [be the first to run BeaverTails](/benchmarks/pku-alignment/beavertails). ## BeaverTails vs Other Benchmarks | Benchmark | What it tests | Categories | Saturation | |---|---|---|---| | **BeaverTails** | Multi-label QA-pair harm classification | 14 | Medium | | Aegis v2 (Response) | Fine-grained taxonomy response classification | 12+9 | Medium | | HarmBench (Response) | Jailbreak success detection | N/A (behavior-based) | Medium | ## Run BeaverTails on Your Model Benchgen tracks version-controlled, regression-tested guard-model performance — submit your classifier's BeaverTails scores to compare against Shieldstral and other guard models. ## Related Models & Benchmarks - Models: [Shieldstral 1.0](/models/mistral-ai/shieldstral-1-0) · [GPT-OSS-Safeguard-20B](/models/openai/gpt-oss-safeguard-20b) - Benchmarks: [Aegis v2 (Response)](/benchmarks/nvidia/aegis-v2-response) · [HarmBench (Response)](/benchmarks/center-for-ai-safety/harmbench-response) ## Frequently Asked Questions

What is BeaverTails?

BeaverTails is a large-scale, human-labeled QA-pair safety dataset spanning 14 harm categories, released by the PKU Alignment team.

Can one example belong to multiple harm categories?

Yes — BeaverTails uses multi-label annotation, so a single QA pair can be tagged with more than one of the 14 categories.

Who created BeaverTails?

Researchers from Peking University's PKU Alignment team (Ji et al., NeurIPS 2023).

--- *Last updated 2026-08-12.*

Rank	Model	Score
1	qwen3guard-8b	85.9
2	shieldstral-1-0	85
3	gpt-oss-safeguard-20b	83.8
4	nemotron-3-5-content-safety-4b	83.3
5	llamaguard-4-12b	69.8
6	shieldgemma-9b	54

BeaverTails — Results

BeaverTails — Evaluations