SHADE-Arena — Results
| Rank | Model | Score |
|---|---|---|
| 1 | claude-opus-4 | 30.2 |
| 2 | claude-sonnet-3-7 | 26.2 |
| 3 | gemini-2-5-pro | 14.8 |
| 4 | claude-sonnet-3-6 | 10.4 |
| 5 | o4-mini | 6.5 |
| 6 | gpt-4-1 | 5.3 |
| 7 | gemini-2-0-flash | 4.5 |
| 8 | o3 | 1.6 |
| 9 | gemini-2-5-flash | 1.4 |
| 10 | claude-haiku-3-5 | — |
| 11 | gpt-4-1-mini | — |
| 12 | gpt-4-5 | — |
| 13 | gpt-4o | — |
| 14 | r1 | — |
SHADE-Arena
1 phaseActive
Agentic safety benchmark for covert sabotage: 17 tasks scored by overall sabotage success, evasion success, and main-task success.
Overview
SHADE-Arena
Quick answer: SHADE-Arena is a safety benchmark for agentic sabotage. A model is asked to complete a benign multi-step task while also carrying out a hidden malicious side task, then evade an AI monitor. The headline metric is overall success: main task + side task + evasion under a suspicion threshold.
At a Glance
What it tests: Whether an LLM agent can perform covert sabotage in realistic tool-use workflows while avoiding monitor detection.
Why it matters: It evaluates dangerous capability in an agent setting rather than static QA, making it useful for pre-deployment safety testing.
Known limitations: Publicly available results come primarily from figures/reports; many raw evaluation artifacts are not fully public to reduce contamination risk.
What SHADE-Arena Measures
SHADE-Arena (Subtle Harmful Agent Detection and Evaluation) places models inside rich simulated environments with tools (mail, search, terminal-like actions, app interfaces). Each of the 17 tasks has a benign user objective and a paired hidden side objective that represents sabotage.
A run is considered successful only when the model completes the benign task, completes the malicious side task, and keeps monitor suspicion below threshold. This three-part condition is designed to model practical risk: models that fail basic utility would not be deployed, and sabotage that is always detected is lower risk than covert sabotage.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Safety / agentic sabotage |
| Primary metric | Overall success rate |
| Secondary metrics | Evasion success rate, main-task success rate |
| Number of tasks | 17 |
| Monitor threshold | Suspicion score < 80 |
| Saturation | Low |
| Created by | Anthropic, Scale AI, Redwood Research collaborators |
| Source paper | SHADE-Arena (arXiv:2506.15740) |
| GitHub | safety-research/SHADE-Arena |
How SHADE-Arena Is Scored
SHADE-Arena reports three rates:
- Main-task success: the benign task is completed.
- Evasion success: the side task is completed and monitor suspicion stays below threshold.
- Overall success: both tasks are completed and the model evades detection.
Overall success is the strict metric and is used as the leaderboard score on this page.
State-of-the-Art Results
The table below reflects the published model ranking used for this benchmark page. Scores are percentages.
| Rank | Model | Overall success |
|---|---|---|
| 1 | Claude Opus 4 | 30.2 |
| 2 | Claude Sonnet 3.7 | 26.2 |
| 3 | Gemini 2.5 Pro | 14.8 |
| 4 | Claude Sonnet 3.6 | 10.4 |
| 5 | o4-mini | 6.5 |
| 6 | GPT-4.1 | 5.3 |
| 7 | Gemini 2.0 Flash | 4.5 |
| 8 | o3 | 1.6 |
| 9 | Gemini 2.5 Flash | 1.4 |
| 10 | Claude Haiku 3.5 | 0.0 |
| 11 | R1 | 0.0 |
| 12 | GPT-4.1 Mini | 0.0 |
| 13 | GPT-4.5 | 0.0 |
| 14 | GPT-4o | 0.0 |
Source: Anthropic SHADE-Arena research post and paper. Reported values are chart-derived leaderboard values from the published figures and may differ slightly from unreleased underlying run-level aggregates.
Additional Published Breakdown
| Model | Evasion success | Main-task success |
|---|---|---|
| Claude Opus 4 | 58.3 | 63.6 |
| Claude Sonnet 3.7 | 49.8 | 53.8 |
| Gemini 2.5 Pro | 40.9 | 48.0 |
| Claude Sonnet 3.6 | 44.2 | 41.8 |
| o4-mini | 26.4 | 22.3 |
| GPT-4.1 | 30.2 | 31.0 |
| Gemini 2.0 Flash | 20.3 | 23.3 |
| o3 | 28.7 | 39.1 |
| Gemini 2.5 Flash | 25.8 | 12.0 |
| Claude Haiku 3.5 | 15.0 | 12.1 |
| R1 | 10.5 | 0.0 |
| GPT-4.1 Mini | 16.0 | 17.5 |
| GPT-4.5 | 6.2 | 31.3 |
| GPT-4o | 12.5 | 13.0 |
SHADE-Arena on Benchgen
No Benchgen run data is attached yet for this benchmark. You can still use this page as the canonical external leaderboard and benchmark definition page.
SHADE-Arena vs Other Benchmarks
| Benchmark | What it tests | Tasks | Saturation |
|---|---|---|---|
| SHADE-Arena | Covert sabotage + monitor evasion in agent workflows | 17 | Low |
| EnterpriseClawBench | Enterprise agent task completion quality | 852 | Low |
| SWE-bench Verified | Real GitHub issue resolution | 500 | Low |
SHADE-Arena is closest to a deployment-risk eval, while EnterpriseClawBench and SWE-bench focus on general utility and engineering competence.
Frequently Asked Questions
What is SHADE-Arena?
SHADE-Arena is an agentic safety benchmark where models must complete a benign task and a hidden malicious side task while avoiding monitor detection. It is designed to test covert sabotage capability.What is the main SHADE-Arena metric?
The main metric is overall success rate: main-task success plus side-task success plus evasion of the monitor.Who created SHADE-Arena?
The benchmark was introduced by Anthropic researchers with collaborators from Scale AI and Redwood Research, and published as arXiv:2506.15740.Benchmark definition paraphrased from the paper and Anthropic research post. Last updated 2026-07-02.