OpenAI Moderation — Results
| Rank | Model | Score |
|---|---|---|
| 1 | gpt-oss-safeguard-20b | 84 |
| 2 | shieldstral-1-0 | 81.4 |
| 3 | shieldgemma-9b | 78.6 |
| 4 | qwen3guard-8b | 74.7 |
| 5 | nemotron-3-5-content-safety-4b | 74.7 |
| 6 | llamaguard-4-12b | 73.9 |
OpenAI Moderation
1 phaseActive
OpenAI's public moderation evaluation dataset. 1,680 labeled examples across 8 categories. Markov et al., 2023.
Overview
OpenAI Moderation
Quick answer: The OpenAI Moderation benchmark evaluates content classifiers against OpenAI's public moderation evaluation dataset — 1,680 examples labeled across 8 harm categories, built from real production API traffic. It's one of the earliest widely-used content moderation benchmarks and remains a common reference point for guard-model comparisons.
At a Glance
What it tests: Multi-category harmful content detection — sexual, hateful, violent, self-harm, and harassment content, among other categories, drawn from real-world usage patterns. Why it matters: The dataset was built from real OpenAI API traffic rather than purely synthetic or crowdsourced data, giving it a production-representative harm distribution that's still referenced nearly four years after release. Known limitations: English-language, single-category-per-example scoring conventions vary by paper; some later works report a coarser binary harmful/not-harmful F1 rather than per-category scores.
What OpenAI Moderation Measures
OpenAI's moderation evaluation dataset contains 1,680 prompts labeled across 8 top-level content categories (e.g., sexual, hate, violence, self-harm, harassment), built to validate OpenAI's production Moderation API. Guard-model papers typically report an aggregate F1 across categories, testing whether a classifier's harm judgments align with OpenAI's human-labeled ground truth.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Safety / content moderation |
| Metric | F1 score (%) |
| Test set size | 1,680 examples |
| Categories | 8 top-level harm categories |
| Created by | Todor Markov, Chong Zhang, Sandhini Agarwal, Florentine Eloundou Nekoul, Theodore Lee, Steven Adler, Angela Jiang, Lilian Weng (OpenAI) |
| Paper | A Holistic Approach to Undesired Content Detection (arXiv 2208.03274) |
| GitHub | openai/moderation-api-release |
How OpenAI Moderation Is Scored
Each of the 1,680 examples' human-labeled harm categories are compared against a classifier's predictions and scored as F1.
State-of-the-Art Results
Scores from the Shieldstral model card (Mistral AI, August 2026). F1 (%).
| Rank | Model | Score | Source | Date |
|---|---|---|---|---|
| 1 | GPT-OSS-Safeguard-20B | 84% | Mistral AI model card | 2026-08 |
| 2 | Shieldstral 1.0 | 81.4% | Mistral AI model card | 2026-08 |
| 3 | ShieldGemma-9B | 78.6% | Mistral AI model card | 2026-08 |
| 4 | Qwen3Guard-8B | 74.7% | Mistral AI model card | 2026-08 |
| 5 | Nemotron-3.5-Content-Safety-4B | 74.7% | Mistral AI model card | 2026-08 |
| 6 | LlamaGuard-4-12B | 73.9% | Mistral AI model card | 2026-08 |
Scores sourced from Mistral AI's published model card, shown for context. Not Benchgen measurements.
OpenAI Moderation on Benchgen
No Benchgen results yet — be the first to run OpenAI Moderation.
OpenAI Moderation vs Other Benchmarks
| Benchmark | What it tests | Test size | Saturation |
|---|---|---|---|
| OpenAI Moderation | Multi-category content moderation | 1,680 | Medium |
| ToxicChat | Real-world chatbot query toxicity | 10,165 | Medium |
| BeaverTails | QA-pair harm classification | ~700 | Medium |
Run OpenAI Moderation on Your Model
Benchgen tracks version-controlled, regression-tested guard-model performance — submit your classifier's OpenAI Moderation scores to compare against Shieldstral and other guard models.
Related Models & Benchmarks
- Models: Shieldstral 1.0 · GPT-OSS-Safeguard-20B
- Benchmarks: ToxicChat · BeaverTails
Frequently Asked Questions
What is the OpenAI Moderation dataset?
It's a public evaluation dataset released by OpenAI in 2022, containing 1,680 examples labeled across 8 harm categories, built from real production API traffic to validate OpenAI's Moderation API.Is this the same as calling OpenAI's live Moderation API?
No — this benchmark uses the static, publicly released evaluation dataset. Scores here reflect third-party classifiers evaluated against that dataset's ground-truth labels, not OpenAI's current production API.Who created this dataset?
OpenAI (Markov et al., published at AAAI 2023).Last updated 2026-08-12.