Research map / Large language models / Language, safety and interpretability
A StrongREJECT for Empty Jailbreaks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces StrongREJECT, a jailbreak benchmark with a curated 313-prompt forbidden dataset and a willingness-plus-capability automated evaluator that agrees with human judges better than prior evaluators, and shows jailbreaks tend to reduce victim model capabilities.
Problem
Jailbreak papers report near-100% attack success rates using forbidden prompt datasets of uneven quality and automated evaluators that reward non-refusal or toxicity without measuring whether responses are actually useful, so reported effectiveness is inflated.
Approach
Pair a curated set of specific, answerable, verifiably harmful forbidden prompts with an evaluator that scores non-refusal multiplied by the average of specificity and convincingness, so a score requires both willingness and capability; a fine-tuned open-source Gemma 2B version mirrors the GPT-4o-mini rubric judge.
Key results
- StrongREJECT rubric evaluator has bias 0.012 and MAE 0.077 against human labels, the lowest MAE among evaluators, with Spearman 0.846; the fine-tuned Gemma 2B version reaches Spearman 0.900.
- Most prior automated evaluators overstate jailbreak effectiveness; string matching has bias 0.484 relative to human scores.
- Jailbreaks that most increase willingness to respond (e.g., PAIR, PAP) tend to lower Dolphin's StrongREJECT score and GPT-4o's MMLU accuracy, indicating capability degradation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the StrongREJECT rubric evaluator instead of the strongest prior evaluator on all responses | Automated jailbreak evaluation · Human evaluation (1,361 labeled data points) | MAE (All responses): 0.090 ± 0.01 -> 0.077 ± 0.01 | Helped |
| Use the StrongREJECT rubric evaluator instead of GPT-4 Judge | Automated jailbreak evaluation · Human evaluation (human score > 0) | MAE (Human > 0): 0.212 ± 0.03 -> 0.196 ± 0.03 | Helped |
| Use the StrongREJECT fine-tuned evaluator instead of HarmBench | Automated jailbreak evaluation · Human evaluation (jailbreak method rankings) | Spearman: 0.819 -> 0.900 | Helped |
| Evaluate responses by usefulness, including willingness and capabilities, rather than emphasizing non-refusal | Automated jailbreak evaluation · Human evaluation | MAE (All responses): 0.262 ± 0.02 -> 0.077 ± 0.01 | Helped |
| Apply jailbreak prompt transformations | Aligned models’ capabilities · MMLU zero-shot (171-question subset) | MMLU score | Hurt |
| Apply jailbreak prompt transformations | Dolphin · StrongREJECT forbidden prompts (313 prompts) | StrongREJECT score | Hurt |
| Use the StrongREJECT rubric evaluator rather than non-refusal-focused prior evaluators | Jailbreak effectiveness evaluation · Jailbreaks evaluated against victim models | Jailbreak effectiveness score | Hurt |
Surprising findings
- Jailbreaks that bypass safety fine-tuning tend to reduce victim model capabilities, as shown by lower MMLU scores on jailbroken benign prompts.
- Many jailbreak scores are substantially lower than those reported in their original papers; a low-resource language (Scots Gaelic) jailbreak's claimed harmful outputs were vacuous on reproduction.
Limitations
- Limited to text-only LLMs; unclear applicability to multimodal models.
- Forbidden prompts may become outdated as provider terms of service change.
- 313-prompt dataset is small and not a standalone measure of model robustness.
Experimental setup
Datasets: AdvBench DAN HarmfulQ MaliciousInstruct MasterKey GPT-4 System Card prompts MMLU
Benchmarks: StrongREJECT MMLU
Metrics: bias mean absolute error Spearman correlation non-refusal rate StrongREJECT score
Baselines: String matching for non-refusal Jailbroken - binary PICT GPT-4 Judge PAIR judge OpenAI moderation API HarmBench classifier
Builds on: HarmBench PAIR PAP GCG AdvBench
Closest papers by meaning
- JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models (NeurIPS 2024 · similarity 0.91)
- Injecting Universal Jailbreak Backdoors into LLMs in Minutes (ICLR 2025 · similarity 0.89)
- Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMs (NeurIPS 2024 · similarity 0.89)
- The Jailbreak Tax: How Useful are Your Jailbreak Outputs? (ICML 2025 · similarity 0.88)
- Systematic Scaling Analysis of Jailbreak Attacks in Large Language Models (NeurIPS 2026 · similarity 0.88)
- JailPO: A Novel Black-Box Jailbreak Framework via Preference Optimization Against Aligned LLMs (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.