Research map / Large language models / Language, safety and interpretability
The Jailbreak Tax: How Useful are Your Jailbreak Outputs?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measures the utility of jailbroken LLM outputs by pseudo-aligning models to refuse benign tasks (biology MCQ, math) and finds a consistent "jailbreak tax" in accuracy across eight jailbreak attacks.
Problem
Jailbreak attacks bypass LLM safety guardrails, but it is unclear whether the harmful outputs they elicit are actually useful, since utility of unsafe answers is hard to evaluate and no unaligned baseline is typically available.
Approach
Align models to refuse benign, easily verifiable tasks (biology MCQ, GSM8K/MATH math) via system prompts, SFT, or harmful rewording (EvilMath), then measure whether jailbreaks restore correct answers relative to an unaligned baseline, defining the jailbreak tax as the relative utility loss.
Key results
- Jailbreak tax is substantial for some attacks: PAIR reduces GSM8K accuracy by up to 91-92% on LLaMA 3.1 70B with system prompt alignment.
- System prompt jailbreak and finetuning and many-shot jailbreaking largely preserve utility, showing the tax is not inherent to alignment.
- Claude 3.5 Haiku on EvilMath: PAIR and TAP exceed 99% success but accuracy drops up to 26%.
- No clear correlation between jailbreak success rate and jailbreak tax; larger 405B model does not reduce the tax.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply the PAIR jailbreak to system-prompt-aligned LLaMA 3.1 70B | LLaMA 3.1 70B · GSM8K | accuracy: up to 91% drop in accuracy | Hurt |
| Apply jailbreak attacks that substantially modify instructions | Aligned models · GSM8K | accuracy: up to a 92% reduction | Hurt |
| Use a system-prompt jailbreak to counter system-prompt alignment | LLaMA 3.1 70B · WMDP and GSM8K | utility | No clear effect |
| Apply the finetuning jailbreak or Many-shot jailbreak | LLaMA 3.1 70B · WMDP and GSM8K | utility | No clear effect |
| Use PAIR instead of attacks with similar jailbreak success | Jailbreak attacks · GSM8K | jailbreak tax | Mixed |
| Increase model size | LLaMA 3.1 · WMDP and GSM8K | jailbreak tax | No clear effect |
| Use PAIR or TAP under SFT alignment | LLaMA 3.1 70B · WMDP and GSM8K | jailbreak tax | Mixed |
| Apply PAIR or TAP to Claude 3.5 Haiku | Claude 3.5 Haiku · EvilMath | accuracy: up to 26% | Hurt |
| Apply PAIR, TAP, or MultiJail to the hardest MATH tasks | LLaMA 3.1 70B with system prompt alignment · MATH level 5 | accuracy: more than 40%; more than 80% | Hurt |
| Increase mathematical task difficulty | LLaMA 3.1 70B with system prompt alignment · GSM8K and MATH levels 1, 3, and 5 | jailbreak tax | No clear effect |
| Align the model using SFT or a system prompt | LLaMA 3.1 8B · MMLU social science and humanities subset | accuracy: 0.8358; 0.8463; 0.8407 | No clear effect |
Surprising findings
- More capable models (LLaMA 3.1 405B) do not reduce the jailbreak tax; the most successful jailbreaks like PAIR incur the highest tax there.
- Harder MATH levels do not systematically increase the jailbreak tax relative to GSM8K.
Limitations
- Uses pseudo-aligned models on benign tasks rather than genuinely harmful tasks, so usefulness for real harm is not directly measured.
- Evaluated on math and biology multiple-choice; generalization to open-ended harmful tasks is unclear.
- Attack hyperparameters and attacker/judge choices (e.g., GPT-4o-mini for PAIR) may affect results.
Experimental setup
Datasets: WMDP GSM8K MATH MMLU EvilMath (constructed from GSM8K) UnicornMath (constructed from GSM8K)
Benchmarks: WMDP-bio GSM8K MATH UnicornMath MMLU
Metrics: jailbreak success rate jailbroken utility jailbreak tax
Baselines: System prompt jailbreak Finetuning (counter-alignment) Many-shot jailbreak GCG AutoDAN MultiJail PAIR TAP
Builds on: GCG PAIR TAP AutoDAN Many-shot jailbreaking StrongREJECT AgentHarm
Closest papers by meaning
- A StrongREJECT for Empty Jailbreaks (NeurIPS 2024 · similarity 0.88)
- JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models (NeurIPS 2024 · similarity 0.87)
- Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMs (NeurIPS 2024 · similarity 0.87)
- Jailbreaking Large Language Models Against Moderation Guardrails via Cipher Characters (NeurIPS 2024 · similarity 0.86)
- Systematic Scaling Analysis of Jailbreak Attacks in Large Language Models (NeurIPS 2026 · similarity 0.86)
- h4rm3l: A Language for Composable Jailbreak Attack Synthesis (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.