Research map / Adversarial robustness and security / Attacks on text and graphs
Fight Back Against Jailbreaking via Prompt Adversarial Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Prompt Adversarial Tuning (PAT), which optimizes a defensive prefix control via alternating adversarial (min-min) GCG-style updates with benign utility loss, to protect LLMs against jailbreaks at inference time.
Problem
Jailbreak attacks make aligned LLMs produce harmful content; existing defenses are either costly fine-tuning or heuristic prompt-based methods, and achieving intrinsic robustness through prompt optimization was open.
Approach
Adversarially trains a short defense control prefix against an attack control (alternating min-min updates with GCG-style token search), jointly optimized with benign QA pairs to preserve utility; the learned prefix is prepended to user prompts at inference and can transfer across models.
Key results
- On Vicuna-7B, PAT reduces average ASR across five attacks to 1.8% (vs 70.8% no defense) while achieving the highest MT-bench score (6.68).
- On Llama-2-7B, PAT achieves average ASR of 0.8% with MT-bench 6.78.
- Transferred PAT prefix on GPT-4 yields 0% ASR for GCG, AutoDAN and ICA and 2% for PAIR and TAP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the PAT defense prefix | Vicuna-7B · AdvBench, average across GCG, AutoDAN, ICA, PAIR, and TAP | Average ASR: 70.8% -> 1.8% | Helped |
| Add the PAT defense prefix | Llama-2-7B · AdvBench, average across GCG, AutoDAN, ICA, PAIR, and TAP | Average ASR: 32.6% -> 0.8% | Helped |
| Add the PAT defense prefix | Vicuna-7B · MT-bench | MT-bench: 6.55 -> 6.68 | Helped |
| Add the PAT defense prefix | Vicuna-7B · MMLU | MMLU: 51.2 -> 50.9 | Hurt |
| Add the PAT defense prefix | GPT-3.5 · Automated jailbreak attacks | ASR: 92% -> 4% | Helped |
| Add the PAT defense prefix | GPT-4 · Automated jailbreak attacks | ASR: 5% -> 0% | Helped |
| Increase defense control length (ablation) | PAT on Vicuna-7B · AdvBench, GCG/AutoDAN/PAIR | ASR | Helped |
| Increase defense control length excessively (ablation) | PAT on Vicuna-7B · Benign utility | MMLU | Hurt |
| Decrease trade-off coefficient α (ablation) | PAT on Vicuna-7B · AdvBench, GCG/AutoDAN/PAIR | ASR | Helped |
| Decrease trade-off coefficient α (ablation) | PAT on Vicuna-7B · Benign utility | MMLU | Hurt |
| Apply PAT under adaptive attack | Vicuna-7B · GCG adaptive attack | ASR: 92% -> 23% | Helped |
| Apply PAT under adaptive attack | Llama-2-7B · GCG adaptive attack | ASR: 36% -> 12% | Helped |
Surprising findings
- PAT raised MT-bench scores above the undefended model, which the authors attribute to improved logical/reasoning ability, though MMLU dipped slightly.
- PPL filtering reduces GCG ASR to 0% on Vicuna-7B but fails against low-perplexity attacks such as AutoDAN and PAIR.
Limitations
- Effectiveness may decrease under adaptive attacks that know the defense, though the paper includes a limited adaptive attack study.
- Robustness-utility trade-off: smaller alpha improves robustness but reduces benign utility; longer control may hurt utility.
- Jailbreak methods evolve rapidly, so defenses may face new threats.
Experimental setup
Datasets: AdvBench MS MARCO
Benchmarks: AdvBench MT-bench MMLU
Metrics: attack success rate MT-bench score MMLU accuracy
Baselines: PPL filtering Self-reminder ICD DRO SafeDecoding SmoothLLM GCG AutoDAN ICA PAIR TAP
Builds on: greedy coordinate gradient adversarial training GCG
Closest papers by meaning
- Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks (NeurIPS 2024 · similarity 0.92)
- Guaranteed Jailbreaking Defense via Disrupt-and-Rectify Smoothing (NeurIPS 2026 · similarity 0.91)
- Short-length Adversarial Training Helps LLMs Defend Long-length Jailbreak Attacks: Theoretical and Empirical Evidence (NeurIPS 2025 · similarity 0.90)
- Improved Generation of Adversarial Examples Against Safety-aligned LLMs (NeurIPS 2024 · similarity 0.90)
- RobustKV: Defending Large Language Models against Jailbreak Attacks via KV Eviction (ICLR 2025 · similarity 0.90)
- AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.