Research map / Adversarial robustness and security / Attacks on text and graphs
Short-length Adversarial Training Helps LLMs Defend Long-length Jailbreak Attacks: Theoretical and Empirical Evidence
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows theoretically (linear transformers on in-context regression) and empirically (AT on five 7-8B LLMs with GCG suffixes) that adversarial training with suffix length ~sqrt(M) defends against suffix jailbreaks of length M, with ASR correlating with sqrt(M_test)/M_train.
Problem
Adversarial training for LLM jailbreak defense typically uses long adversarial suffixes, which are expensive to synthesize; it is unclear how the AT suffix length affects robustness to attacks with other suffix lengths.
Approach
Model the problem as adversarial in-context learning on linear self-attention models with a suffix attack, prove a robust generalization bound scaling with sqrt(M_test)/M_train, and test the implied short-AT-defends-long-attack relationship on real LLMs with GCG-based adversarial training.
Key results
- Pearson correlations above 0.7 with p<0.05 between sqrt(M_test)/M_train and GCG ASR on Vicuna-7B, Mistral-7B, Llama-2-7B, Llama-3-8B and Qwen2.5-7B.
- AT with adversarial suffix length 20 reduces ASR of suffix attacks with lengths up to 120 by at least 30% in all experiments.
- DeepInception ASR drops to 0% with M_train=20 AT on Vicuna-7B and Qwen2.5-7B; PAIR ASR drops from 84% to about 50% on Vicuna-7B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with GCG adversarial suffixes of length 20 (ablation) | LLMs evaluated against suffix jailbreak attacks · Suffix jailbreak attacks with suffix length up to 120 | ASR: at least 30% | Helped |
| Increase adversarial suffix length during AT (ablation) | LLMs evaluated against suffix jailbreak attacks · Suffix jailbreak attacks | ASR | Helped |
| Use GCG AT; correlate test suffix length with AT suffix length as √M_test/M_train | Vicuna-7B, Mistral-7B, Llama-2-7B, Llama-3-8B, and Qwen2.5-7B · GCG suffix attacks | ASR correlation with √M_test/M_train: 0.93; 0.86; 0.88; 0.76; 0.87 | Mixed |
| Use single-attack GCG AT rather than training on multiple attacks | LLMs evaluated against unseen jailbreak attacks · BEAST and GCQ | ASR correlation with √M_test/M_train | Mixed |
| Apply short-length adversarial training (ablation) | Vicuna-7B · PAIR | ASR (%): 84.0 -> around 50% | Helped |
| Apply short-length adversarial training (ablation) | Qwen2.5-7B · PAIR | ASR (%): 71.0 -> around 25% | Helped |
| Further increase AT suffix length after short-length training (ablation) | LLMs evaluated against PAIR · PAIR | ASR | No clear effect |
| Use short-length adversarial training (ablation) | Vicuna-7B and Qwen2.5-7B · DeepInception | ASR (%): 76.0 -> 0.0; 89.0 -> 0.0 | Helped |
| Apply adversarial training (ablation) | Adversarially trained LLMs · AlpacaEval2 | LC-WinRate | Hurt |
| Increase adversarial suffix length during AT from 20 or 30 to 50 (ablation) | LLM adversarial training · AT runtime | time cost: around 30% to 60% longer | Hurt |
Surprising findings
- Correlation with the ratio does not hold for AmpleGCG and Zhu's AutoDAN, because AT already drives their ASR to near zero.
- Increasing the AT suffix length beyond 20 does not substantially help against PAIR.
Limitations
- AT uses only a single attack (GCG), and correlation with unseen attacks (BEAST, GCQ) is significant only for some models.
- Theory analyzes linear self-attention on linear regression with Gaussian data and gradient flow, a considerable gap from real LLMs.
Experimental setup
Datasets: HarmBench Alpaca AdvBench AlpacaEval
Benchmarks: HarmBench test subset AdvBench subset AlpacaEval2
Metrics: attack success rate Pearson correlation coefficient length-controlled win rate
Baselines: GCG BEAST AmpleGCG AutoDAN GCQ PAIR DeepInception
Builds on: GCG in-context learning theory of linear self-attention LoRA
Closest papers by meaning
- Adversarial Reasoning at Jailbreaking Time (ICML 2025 · similarity 0.91)
- Improved Generation of Adversarial Examples Against Safety-aligned LLMs (NeurIPS 2024 · similarity 0.91)
- Fight Back Against Jailbreaking via Prompt Adversarial Tuning (NeurIPS 2024 · similarity 0.90)
- GASP: Efficient Black-Box Generation of Adversarial Suffixes for Jailbreaking LLMs (NeurIPS 2025 · similarity 0.89)
- Efficient Adversarial Training in LLMs with Continuous Attacks (NeurIPS 2024 · similarity 0.89)
- Understanding and Enhancing the Transferability of Jailbreaking Attacks (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.