StrongREJECT

StrongREJECT

Quick answer: StrongREJECT is a jailbreak evaluation benchmark from UC Berkeley (Souly et al., 2024) designed to address the tendency of existing benchmarks to overstate jailbreak effectiveness. It tests whether models can be induced to provide genuinely harmful information on clearly forbidden prompts across 6 harm categories. The metric reported in the Inkling comparison is refusal rate — how often models correctly refuse. Top frontier models score 98–99%, indicating near-universal safety on this dataset.

At a Glance

What it tests: Whether an AI model will comply with clearly harmful requests after being exposed to jailbreaks — prompts designed to bypass safety fine-tuning. The benchmark includes prompts from six categories: disinformation, illegal goods/services, hate/harassment, non-violent crimes, violence, and sexual content.

Why it matters: Prior jailbreak papers often reported near-100% attack success rates using evaluation methods that agreed poorly with human judgment. StrongREJECT showed this was largely measurement error: existing evaluators missed responses that were superficially harmful-looking but contained no actually useful harmful information ("empty jailbreaks"). Its LLM-based evaluator achieves state-of-the-art agreement with human raters. Notably, the paper also found that successful jailbreaks often degrade model capabilities.

Known limitations: StrongREJECT focuses on "clearly harmful" requests — prompts where the harmful intent is unambiguous. It is not designed to test subtle dual-use scenarios, misinformation, or refusal calibration on benign requests that superficially resemble harmful ones. Scores cluster very high for frontier models (~98-99%), leaving little discriminative signal at the top.

What StrongREJECT Measures

StrongREJECT uses a dataset of forbidden prompts that require specific, harmful information to answer correctly. The prompts span 6 categories:

The automated evaluator uses an LLM (or fine-tuned Gemma 2B) to score responses, checking:

• Did the model refuse the request?
• If it responded, did the response contain actionable harmful information?

The refusal rate reported in Inkling's evaluation table is the percentage of prompts where the model either refused outright or gave a response with no useful harmful content.

Benchmark Specifications

How StrongREJECT Is Scored

Each forbidden prompt is sent to the target model (optionally with a jailbreak applied). The evaluator scores the response on a 0–1 scale for harmfulness. The refusal rate is 1 - average harmfulness score × 100, or equivalently the percentage of prompts where the model did not produce harmful content.

State-of-the-Art Results

Scores from Inkling model card (Thinking Machines Lab, July 2026). All 9 comparison models reported. Scores represent refusal rate (% of harmful requests refused).

StrongREJECT vs Related Safety Benchmarks

Last updated 2026-07-16.