Research map / Adversarial robustness and security / Attacks on text and graphs
Efficient Adversarial Training in LLMs with Continuous Attacks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CAT and CAPO, efficient adversarial training for LLMs using continuous embedding-space attacks, showing robustness extrapolates to discrete attacks (GCG, AutoDAN, PAIR) at ~299x lower compute than R2D2.
Problem
Adversarial training for LLMs with discrete attacks (e.g., GCG) is computationally prohibitive, and existing methods like R2D2 trade off utility and robustness poorly.
Approach
Replace discrete suffix attacks in adversarial training with continuous embedding-space attacks, combined with an unlikelihood loss (CAT) or an adversarial IPO loss that needs no utility data (CAPO).
Key results
- CAT and CAPO reduce GCG, AutoDAN and PAIR attack success rates substantially; Phi-3-CAPO reaches 0% ASR on GCG, AutoDAN and PAIR.
- Continuous AT is about 299x cheaper than R2D2 in total forward/backward passes (234,000 vs 165,632,000 for CAT).
- Zephyr + R2D2 refuses 308 benign MMLU questions under chat template, while CAT models show far fewer refusals.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace discrete R2D2 attacks with continuous attacks in CAT/CAPO | Adversarial training on Mistral · Training compute | F/B (total): 165,632,000 -> 234,000 (CAT); 165,632,000 -> 552,960 (CAPO) | Helped |
| Train Phi-3-mini with CAT | Phi-3-mini · HarmBench test | GCG ASR (%): 25 -> 5 | Helped |
| Train Phi-3-mini with CAPO instead of CAT | Phi-3-mini · HarmBench test | PAIR ASR (%): 40 -> 0 | Helped |
| Train Gemma-2B-IT with CAT | Gemma-2B-IT · HarmBench test | GCG ASR (%): 70 -> 5 | Helped |
| Train Mistral with CAT | Mistral · HarmBench test | PAIR ASR (%): 90.0 -> 77.5 | Helped |
| Train Zephyr-7B with CAPO | Zephyr-7B · HarmBench test | PAIR ASR (%): 87.5 -> 10 | Helped |
| Use CAT instead of R2D2 | Zephyr-7B · Harmless | answered queries (%): 42.5 -> 100 | Helped |
| Remove adversarial attacks during CAPO training (ablation) | Gemma-2B · GCG | ASR difference: -2.5 | No clear effect |
| Reduce CAPO attack iterations to one (ablation) | Gemma-2B · GCG | ASR difference: -62.5 | Helped |
| Vary epsilon from 0.0125 to 0.1 (ablation) | Gemma-IPO with CAPO · GCG loss | loss: approximately 14.9 -> near 0 | Helped |
| Vary IPO beta from 0 to 0.5 (ablation) | Gemma-IPO with CAPO · GCG loss | loss: 6.1 -> 0.8 | Helped |
| Use log(1 - harmful-output probability) instead of negative log harmful-output probability for away loss (ablation) | CAPO · Robustness-utility trade-off | trade-off | Hurt |
Surprising findings
- Zephyr + R2D2 overfits the safety objective and refuses benign prompts, and its utility under chat template is misleadingly high in standard benchmarks.
- Training with IPO and no attacks yields no robustness gains, showing the continuous attacks are essential.
Limitations
- Requires careful hyperparameter selection and many hyperparameters.
- CAT needs a utility dataset that may shift predictions on unrelated tasks.
- Experiments limited to models up to 7B; not tested at 70B scale.
Experimental setup
Datasets: HarmBench UltraChat200k MMLU ARC-E ARC-C MT-Bench
Benchmarks: HarmBench MMLU ARC-E ARC-C MT-Bench Harmless PoliteHarmbench
Metrics: attack success rate accuracy MT-Bench score refusal count GCG loss
Baselines: R2D2 Zephyr-7B NPO Llama-2-7B-chat Mistral-7B-Instruct-v0.1
Builds on: R2D2 IPO DPO unlikelihood training HarmBench
Closest papers by meaning
- MixAT: Combining Continuous and Discrete Adversarial Training for LLMs (NeurIPS 2025 · similarity 0.91)
- Robust LLM safeguarding via refusal feature adversarial training (ICLR 2025 · similarity 0.90)
- Short-length Adversarial Training Helps LLMs Defend Long-length Jailbreak Attacks: Theoretical and Empirical Evidence (NeurIPS 2025 · similarity 0.89)
- Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Space (NeurIPS 2024 · similarity 0.88)
- Statistical Estimation of Adversarial Risk in Large Language Models under Best-of-N Sampling (NeurIPS 2026 · similarity 0.88)
- Scaling Trends in Language Model Robustness (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.