Research map / Adversarial robustness and security / Attacks on text and graphs
Jailbreaking as a Reward Misspecification Problem
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Attributes LLM jailbreak vulnerability to reward misspecification, defines a reward-gap metric (ReGap) using implicit rewards, and trains a suffix generator (ReMiss) that finds reward-misspecified adversarial suffixes.
Problem
Aligned LLMs remain vulnerable to jailbreak attacks, and target-loss-based optimization attacks are a weak proxy for successful jailbreaks; the paper asks why alignment fails and how to find such failures automatically.
Approach
Treat jailbreaking as search for prompts where the implicit reward (log-ratio against a reference model) misranks a harmless response above a harmful one; the reward gap ReGap is minimized, with a reference-model perplexity regularizer, and a generator is finetuned to produce such suffixes at scale.
Key results
- ReMiss reaches test ASR@10 of 94.2% on Vicuna-13b-v1.5 and 98.1% on Vicuna-7b-v1.5 on AdvBench, versus 81.2% for GCG and 78.4% for AutoDAN on the best baseline per model.
- ReGap achieves AUROC 0.97 for detecting successful jailbreaks versus 0.82 for target loss on Vicuna-13b-v1.5.
- ReMiss achieves 100% ASR@10 on GPT-3.5-turbo with transferred suffixes and 93.4% ASR@10 on HarmBench for Vicuna-13b-v1.5 (vs 69.7% for AdvPrompter).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ReMiss reward-gap optimization instead of AdvPrompter's target-based objective | Vicuna-13b-v1.5 · AdvBench test | ASR@10 (%): 67.5 -> 94.2 | Helped |
| Use ReMiss reward-gap optimization instead of AutoDAN's target-based objective | Vicuna-7b-v1.5 · AdvBench test | ASR@1 (%): 63.2 -> 49.0 | Hurt |
| Use ReMiss reward-gap optimization instead of AdvPrompter's target-based objective | Llama2-7b-chat · AdvBench test | ASR@10 (%): 7.7 -> 10.6 | Helped |
| Use ReMiss reward-gap optimization instead of GCG's target-based objective | Mistral-7b-instruct · AdvBench test | ASR@10 (%): 99.0 -> 100.0 | Helped |
| Use ReMiss reward-gap optimization instead of AdvPrompter's target-based objective | Llama3.1-8b-instruct · AdvBench test | ASR@10 (%): 27.9 -> 40.4 | Helped |
| Transfer ReMiss-generated attacks instead of AdvPrompter attacks | GPT-4 · Transferred attacks | ASR@1 (%): 3.1% vs. 22.1% | Helped |
| Use ReMiss-generated attacks instead of AdvPrompter attacks | Vicuna-13b-v1.5 · HarmBench | ASR@10 (%): 69.7% vs. 93.4% | Helped |
| Use ReMiss-generated attacks instead of AdvPrompter attacks | Llama2-7b-chat · HarmBench | ASR@10 (%): 31.5% vs. 37.2% | Helped |
| Use ReGap rather than target loss to identify successful jailbreaks | Vicuna-13b-v1.5 · AdvBench test | AUROC: 0.82 -> 0.97 | Helped |
| Use ReMiss instead of AdvPrompter-generated suffixes | Vicuna-7b-v1.5 · AdvBench test | Readability score: 44.13 -> 58.57 | Helped |
| Use ReMiss instead of AdvPrompter-generated suffixes | Llama2-7b-chat · AdvBench test | Readability score: 7.07 -> 24.04 | Helped |
| Replace the reference model with TinyLlama-1.1b (ablation) | ReMiss on Llama2-7b-chat · AdvBench test | ASR@10 (%): 10.6 -> 10.6 | No clear effect |
Surprising findings
- Removing the reference model (maximizing harmful and minimizing harmless likelihood directly) drops Llama2-7b-chat ASR@10 to 0.0%, showing the reference-model regularizer is critical.
- Llama2-7b-chat remains robust: its reward gaps stay positive even with adversarial suffixes, and ReMiss achieves only 10.6% test ASR@10 on it.
Limitations
- Llama2-7b-chat is largely resistant (test ASR@10 10.6%), and performance on strongly guarded models is limited.
- Gray-box setting requires white-box access to a reference model and target log probabilities, and closed-source transfer is only tested with suffixes from one Llama2-7b attacker.
Experimental setup
Datasets: AdvBench HarmBench RLHF trojan competition poisoned models data
Benchmarks: AdvBench HarmBench
Metrics: ASR@1 ASR@10 perplexity AUROC readability score misspecification rate
Baselines: AdvPrompter AutoDAN GCG
Builds on: AdvPrompter GCG DPO AutoDAN
Closest papers by meaning
- JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models (NeurIPS 2024 · similarity 0.89)
- Fail-Closed Alignment for Large Language Models (NeurIPS 2026 · similarity 0.89)
- Improving LLM Safety Alignment with Dual-Objective Optimization (ICML 2025 · similarity 0.89)
- Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks (NeurIPS 2024 · similarity 0.88)
- Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation (ICLR 2024 · similarity 0.88)
- Safety Alignment Should be Made More Than Just a Few Tokens Deep (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.