Research map / Adversarial robustness and security / Attacks on text and graphs
JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces JailbreakBench, an open benchmark with a jailbreak artifact repository, 100-behavior JBB-Behaviors dataset, standardized red-teaming/defense evaluation framework, and leaderboard, and selects Llama-3-70B as the jailbreak judge.
Problem
Jailbreak attack evaluation lacks a standard practice: costs and success rates are computed inconsistently, many works are not reproducible (withheld prompts, closed code, evolving APIs), and judges disagree.
Approach
An open, community-extensible benchmark that archives jailbreak artifacts, standardizes decoding, threat models and judging, and tracks attacks and test-time defenses on a leaderboard, with a human-validated judge comparison selecting an open-weight Llama-3-70B judge.
Key results
- Llama-3-70B judge achieves 90.7% agreement with expert majority labels with 5.5% FNR, comparable to GPT-4 (90.3%), while being open-weight.
- Prompt with RS achieves 90% attack success on Llama-2 and 78% on GPT-4, while PAIR reaches 69% on Vicuna and 71% on GPT-3.5.
- GCG attack success drops to 3% on Llama-2 and 4% on GPT-4 under the benchmark judge, and Erase-and-Check is the most robust defense among those tested.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Prompt with RS instead of PAIR | jailbreak attacks · JBB-Behaviors, Llama-2 | attack success rate: 0% -> 90% | Helped |
| Use Prompt with RS instead of PAIR | jailbreak attacks · JBB-Behaviors, GPT-4 | attack success rate: 34% -> 78% | Helped |
| Use JB-Chat instead of PAIR | jailbreak attacks · JBB-Behaviors, Vicuna | attack success rate: 69% -> 90% | Helped |
| Use JB-Chat instead of PAIR | jailbreak attacks · JBB-Behaviors, GPT-3.5 | attack success rate: 71% -> 0% | Hurt |
| Apply SmoothLLM to transfer attacks | PAIR attacks on GPT-3.5 · JBB-Behaviors, GPT-3.5 | attack success rate: 71% -> 5% | Helped |
| Apply SmoothLLM to transfer attacks | PAIR attacks on Vicuna · JBB-Behaviors, Vicuna | attack success rate: 69% -> 55% | Helped |
| Apply Perplexity Filter to transfer attacks | PAIR attacks on GPT-3.5 · JBB-Behaviors, GPT-3.5 | attack success rate: 71% -> 17% | Helped |
| Apply Perplexity Filter to transfer attacks | PAIR attacks on Vicuna · JBB-Behaviors, Vicuna | attack success rate: 69% -> 69% | No clear effect |
| Apply Erase-and-Check to transfer attacks | PAIR attacks on GPT-4 · JBB-Behaviors, GPT-4 | attack success rate: 34% -> 1% | Helped |
| Use Llama-3-70B instead of the rule-based judge | Jailbreak judging · JBB-Behaviors judge comparison | agreement: 56.0% -> 90.7% | Helped |
| Use Llama-3-70B instead of GPT-4 as judge | Jailbreak judging · JBB-Behaviors judge comparison | agreement: 90.3% -> 90.7% | No clear effect |
| Use Llama-3-70B instead of Llama Guard as judge | Jailbreak judging · JBB-Behaviors judge comparison | FNR: 60.9% -> 5.5% | Helped |
Surprising findings
- Synonym Substitution defense is surprisingly effective, limiting the best precomputed-jailbreak success rate to 24%.
- The Perplexity Filter is effective only against GCG, and GCG success on GPT models fell to about 5% after safety patches.
Limitations
- Attackers cannot modify the system prompt or prefill responses in the current threat model.
- Only text modality is supported; multimodal jailbreaks are not covered.
- Judge agreement and refusal assessments rely on a relatively small labeled set of 300 examples and some borderline benign behaviors.
Experimental setup
Datasets: JBB-Behaviors AdvBench TDC/HarmBench XS-Test
Benchmarks: JBB-Behaviors XS-Test
Metrics: attack success rate average queries average tokens agreement false positive rate false negative rate refusal rate
Baselines: GCG PAIR JB-Chat Prompt with RS SmoothLLM Perplexity Filter Erase-and-Check GPT-4 judge HarmBench judge Llama Guard Llama Guard 2
Builds on: RobustBench HarmBench AdvBench GCG PAIR
Closest papers by meaning
- Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMs (NeurIPS 2024 · similarity 0.93)
- JailPO: A Novel Black-Box Jailbreak Framework via Preference Optimization Against Aligned LLMs (AAAI 2025 · similarity 0.92)
- Understanding and Enhancing the Transferability of Jailbreaking Attacks (ICLR 2025 · similarity 0.92)
- MMJ-Bench: A Comprehensive Study on Jailbreak Attacks and Defenses for Vision Language Models (AAAI 2025 · similarity 0.92)
- Gradient Cuff: Detecting Jailbreak Attacks on Large Language Models by Exploring Refusal Loss Landscapes (NeurIPS 2024 · similarity 0.91)
- MT-JailBench: A Modular Benchmark for Understanding Multi-Turn Jailbreak Attacks (NeurIPS 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.