# SHADE-Arena

![Category](https://img.shields.io/badge/Category-Safety-1f6feb) ![Metric](https://img.shields.io/badge/Metric-Overall%20Success%20Rate-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-17-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-Low-d29922) ![Created](https://img.shields.io/badge/Created-Jun%202025-555) [![Paper](https://img.shields.io/badge/Paper-arXiv%202506.15740-b31b1b)](https://arxiv.org/abs/2506.15740) [![Research post](https://img.shields.io/badge/Source-Anthropic%20Research-0969da)](https://www.anthropic.com/research/shade-arena-sabotage-monitoring) [![GitHub](https://img.shields.io/badge/GitHub-safety--research%2FSHADE--Arena-181717?logo=github&logoColor=white)](https://github.com/safety-research/SHADE-Arena) > **Quick answer:** SHADE-Arena is a safety benchmark for agentic sabotage. A model is asked to complete a benign multi-step task while also carrying out a hidden malicious side task, then evade an AI monitor. The headline metric is overall success: main task + side task + evasion under a suspicion threshold. ## At a Glance **What it tests:** Whether an LLM agent can perform covert sabotage in realistic tool-use workflows while avoiding monitor detection. **Why it matters:** It evaluates dangerous capability in an agent setting rather than static QA, making it useful for pre-deployment safety testing. **Known limitations:** Publicly available results come primarily from figures/reports; many raw evaluation artifacts are not fully public to reduce contamination risk. ## What SHADE-Arena Measures SHADE-Arena (Subtle Harmful Agent Detection and Evaluation) places models inside rich simulated environments with tools (mail, search, terminal-like actions, app interfaces). Each of the 17 tasks has a benign user objective and a paired hidden side objective that represents sabotage. A run is considered successful only when the model completes the benign task, completes the malicious side task, and keeps monitor suspicion below threshold. This three-part condition is designed to model practical risk: models that fail basic utility would not be deployed, and sabotage that is always detected is lower risk than covert sabotage. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Safety / agentic sabotage | | Primary metric | Overall success rate | | Secondary metrics | Evasion success rate, main-task success rate | | Number of tasks | 17 | | Monitor threshold | Suspicion score < 80 | | Saturation | Low | | Created by | Anthropic, Scale AI, Redwood Research collaborators | | Source paper | [SHADE-Arena (arXiv:2506.15740)](https://arxiv.org/abs/2506.15740) | | GitHub | [safety-research/SHADE-Arena](https://github.com/safety-research/SHADE-Arena) | ## How SHADE-Arena Is Scored SHADE-Arena reports three rates: - **Main-task success:** the benign task is completed. - **Evasion success:** the side task is completed and monitor suspicion stays below threshold. - **Overall success:** both tasks are completed and the model evades detection. Overall success is the strict metric and is used as the leaderboard score on this page. ## State-of-the-Art Results The table below reflects the published model ranking used for this benchmark page. Scores are percentages. | Rank | Model | Overall success | |---|---|---| | 1 | [Claude Opus 4](/models/anthropic/claude-opus-4) | 30.2 | | 2 | [Claude Sonnet 3.7](/models/anthropic/claude-sonnet-3-7) | 26.2 | | 3 | [Gemini 2.5 Pro](/models/google/gemini-2-5-pro) | 14.8 | | 4 | [Claude Sonnet 3.6](/models/anthropic/claude-sonnet-3-6) | 10.4 | | 5 | [o4-mini](/models/openai/o4-mini) | 6.5 | | 6 | [GPT-4.1](/models/openai/gpt-4-1) | 5.3 | | 7 | [Gemini 2.0 Flash](/models/google/gemini-2-0-flash) | 4.5 | | 8 | [o3](/models/openai/o3) | 1.6 | | 9 | [Gemini 2.5 Flash](/models/google/gemini-2-5-flash) | 1.4 | | 10 | [Claude Haiku 3.5](/models/anthropic/claude-haiku-3-5) | 0.0 | | 11 | [R1](/models/deepseek/r1) | 0.0 | | 12 | [GPT-4.1 Mini](/models/openai/gpt-4-1-mini) | 0.0 | | 13 | [GPT-4.5](/models/openai/gpt-4-5) | 0.0 | | 14 | [GPT-4o](/models/openai/gpt-4o) | 0.0 | *Source: Anthropic SHADE-Arena research post and paper. Reported values are chart-derived leaderboard values from the published figures and may differ slightly from unreleased underlying run-level aggregates.* ## Additional Published Breakdown | Model | Evasion success | Main-task success | |---|---:|---:| | Claude Opus 4 | 58.3 | 63.6 | | Claude Sonnet 3.7 | 49.8 | 53.8 | | Gemini 2.5 Pro | 40.9 | 48.0 | | Claude Sonnet 3.6 | 44.2 | 41.8 | | o4-mini | 26.4 | 22.3 | | GPT-4.1 | 30.2 | 31.0 | | Gemini 2.0 Flash | 20.3 | 23.3 | | o3 | 28.7 | 39.1 | | Gemini 2.5 Flash | 25.8 | 12.0 | | Claude Haiku 3.5 | 15.0 | 12.1 | | R1 | 10.5 | 0.0 | | GPT-4.1 Mini | 16.0 | 17.5 | | GPT-4.5 | 6.2 | 31.3 | | GPT-4o | 12.5 | 13.0 | ## SHADE-Arena on Benchgen > No Benchgen run data is attached yet for this benchmark. You can still use this page as the canonical external leaderboard and benchmark definition page. ## SHADE-Arena vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **SHADE-Arena** | Covert sabotage + monitor evasion in agent workflows | 17 | Low | | [EnterpriseClawBench](/benchmarks/enterprise-claw-bench) | Enterprise agent task completion quality | 852 | Low | | [SWE-bench Verified](/benchmarks/swe-bench-verified) | Real GitHub issue resolution | 500 | Low | SHADE-Arena is closest to a deployment-risk eval, while EnterpriseClawBench and SWE-bench focus on general utility and engineering competence. ## Frequently Asked Questions

What is SHADE-Arena?

SHADE-Arena is an agentic safety benchmark where models must complete a benign task and a hidden malicious side task while avoiding monitor detection. It is designed to test covert sabotage capability.

What is the main SHADE-Arena metric?

The main metric is overall success rate: main-task success plus side-task success plus evasion of the monitor.

Who created SHADE-Arena?

The benchmark was introduced by Anthropic researchers with collaborators from Scale AI and Redwood Research, and published as arXiv:2506.15740.

--- *Benchmark definition paraphrased from the paper and Anthropic research post. Last updated 2026-07-02.*

Rank	Model	Score
1	claude-opus-4	30.2
2	claude-sonnet-3-7	26.2
3	gemini-2-5-pro	14.8
4	claude-sonnet-3-6	10.4
5	o4-mini	6.5
6	gpt-4-1	5.3
7	gemini-2-0-flash	4.5
8	o3	1.6
9	gemini-2-5-flash	1.4
10	claude-haiku-3-5	—
11	gpt-4-1-mini	—
12	gpt-4-5	—
13	gpt-4o	—
14	r1	—

SHADE-Arena — Results

SHADE-Arena — Phases

SHADE-Arena