Research map / Adversarial robustness and security / Attacks on text and graphs
Robust LLM safeguarding via refusal feature adversarial training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that jailbreak attacks work largely by ablating a refusal feature in the residual stream, and proposes ReFAT, an efficient adversarial training method that simulates these attacks by ablating the refusal feature during LoRA safety fine-tuning.
Problem
LLMs are vulnerable to diverse adversarial jailbreak attacks, and effective adversarial training defenses are computationally expensive because they require dynamic attack search during fine-tuning.
Approach
Adversarial jailbreaks share a common mechanism of ablating the refusal feature (difference-in-means direction between harmful and harmless activations); RFA approximates worst-case residual-stream perturbations, so training with RFA applied with probability p=0.5 simulates attacks using only extra forward passes instead of gradient-based search.
Key results
- ReFAT reduces RFA attack success rate from 53.3% to 10.4% on Llama3-8B and from 92.8% to 18.8% on Gemma-7B.
- ReFAT is reported as about 1,700x cheaper than R2D2 and 10x cheaper than CAT/LAT in total forward and backward passes, using about 1.5 forward and 1 backward pass per step.
- ReFAT keeps general capability close to baseline, e.g. Llama3-8B MMLU 65.2 vs 65.9 and XSTest 97.6 vs 97.2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with refusal feature ablation (ReFAT) instead of standard refusal training | Llama-3-8B · HarmBench, RFA attack | ASR: 53.3 -> 10.4 | Helped |
| Train with refusal feature ablation (ReFAT) instead of standard refusal training | Gemma-7B · HarmBench, RFA attack | ASR: 92.8 -> 18.8 | Helped |
| Use ReFAT instead of the original model | Llama-3-8B · HarmBench, PAIR attack | ASR: 29.9 -> 11.5 | Helped |
| Use ReFAT instead of the original model | Mistral-7B · HarmBench, GCG attack | ASR: 63.5 -> 16.6 | Helped |
| Use ReFAT instead of the original model | Gemma-7B · HarmBench, AutoDAN attack | ASR: 13.0 -> 6.54 | Helped |
| Use ReFAT instead of the original model | Llama-3-8B · MMLU | score: 65.9 -> 65.2 | No clear effect |
| Use ReFAT instead of the original model | Llama-3-8B · MT-Bench | score: 7.24 -> 6.98 | No clear effect |
| Use ReFAT instead of the original model | Mistral-7B · HarmBench, AutoDAN attack | ASR: 77.4 -> 28.5 | Helped |
| Use ReFAT instead of the original model | Gemma-7B · HarmBench, GCG attack | ASR: 61.4 -> 11.6 | Helped |
| Remove refusal feature ablation during training (RT, p_RFA=0) (ablation) | ReFAT · HarmBench, robustness across attacks | ASR | Hurt |
| Train for more than 1 epoch (ablation) | ReFAT · HarmBench | average ASR | Hurt |
| Ablate refusal features on all layers rather than last 75% or last 50% (ablation) | ReFAT · HarmBench | average ASR | Mixed |
Surprising findings
- Zeroing out the refusal direction degraded performance on Llama-3, so the harmless mean activation is retained in the ablation rather than setting it to zero.
- Refusal training alone often fails to improve and sometimes hurts robustness against attacks (e.g., Gemma-7B GCG ASR rises from 61.4 to 54.7 only slightly, while Mistral-7B RT raises RFA ASR from 85.6 to 70.0 only modestly, and Llama3-8B RT raises AutoDAN ASR relative to base).
Limitations
- ReFAT fails against GCG attacks that switch the model to non-English or colloquial vernacular, since refusal features are computed from English Alpaca instructions.
- Gemma-7B could not be trained with R2D2 due to memory and instability issues, leaving that comparison incomplete.
- Hyperparameter search for baselines was limited to a 40-example subset due to attack cost, so baselines may be under-tuned.
Experimental setup
Datasets: AdvBench Alpaca HarmBench UltraChat XSTest MMLU MT-Bench
Benchmarks: HarmBench AdvBench MMLU MT-Bench XSTest
Metrics: attack success rate MMLU accuracy MT-Bench score XSTest compliance rate forward/backward pass count
Baselines: Refusal training (RT) R2D2 CAT LAT
Builds on: Refusal direction (Arditi et al.) R2D2 Continuous Adversarial Training (CAT) Latent Adversarial Training (LAT) GCG
Closest papers by meaning
- Efficient Adversarial Training in LLMs with Continuous Attacks (NeurIPS 2024 · similarity 0.90)
- Improving LLM Safety Alignment with Dual-Objective Optimization (ICML 2025 · similarity 0.90)
- Short-length Adversarial Training Helps LLMs Defend Long-length Jailbreak Attacks: Theoretical and Empirical Evidence (NeurIPS 2025 · similarity 0.88)
- Safeguarding LLMs via Model-Agnostic Latent Safety Signals from Dark Knowledge (NeurIPS 2026 · similarity 0.88)
- Fight Back Against Jailbreaking via Prompt Adversarial Tuning (NeurIPS 2024 · similarity 0.88)
- Fail-Closed Alignment for Large Language Models (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.