Research map / Large language models / Instruction and fine-tuning
Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attack
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Vaccine, an alignment-stage perturbation-aware (FGSM/SAM-like minimax) training method that makes hidden embeddings robust to harmful embedding drift during user fine-tuning, without access to user data.
Problem
Harmful fine-tuning-as-a-service: a few harmful examples mixed into user fine-tuning data break the safety alignment of LLMs, and defenses requiring user data or heavy per-request computation are impractical.
Approach
Identify harmful embedding drift as the cause of alignment breakage, then during alignment add a worst-case L2-bounded perturbation to each layer's hidden embeddings (computed from a first forward/backward pass) and optimize weights under that perturbation in a second pass, so embeddings remain invariant to later harmful fine-tuning.
Key results
- Vaccine reduces harmful score by up to 9.8% relative to SFT on Llama2-7B across harmful ratios (average HS 46.44 vs 53.24 for SFT) while keeping fine-tune accuracy close (93.4 vs 94.4).
- Under the default setting, Vaccine lowers AlpacaEval harmful score by 13.8% versus SFT while improving fine-tune win rate (44.23 vs 43.14).
- Vaccine roughly doubles alignment training time (0.73 s vs 0.37 s per step on Llama2-7B) with about 0.11 GB extra memory; accelerated variant perturbing every 100 steps adds only marginal harm (+0.60).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply Vaccine alignment instead of SFT alignment | Llama2-7B · SST2 | harmful score: 52.80 -> 42.80 | Helped |
| apply Vaccine alignment instead of SFT alignment | Llama2-7B · AGNEWS | harmful score: 52.60 -> 41.60 | Helped |
| apply Vaccine alignment instead of SFT alignment | Llama2-7B · GSM8K | fine-tune accuracy: 23.40 -> 22.40 | Hurt |
| apply Vaccine alignment instead of SFT alignment | Llama2-7B · AlpacaEval | fine-tune accuracy: 43.14 -> 44.23 | Helped |
| increase perturbation intensity from rho=0.01 to rho=10 (ablation) | Vaccine · default fine-tuning setting | harmful score: 54.40 -> 44.20 | Helped |
| increase perturbation intensity from rho=0.01 to rho=10 (ablation) | Vaccine · default fine-tuning setting | fine-tune accuracy: 94.40 -> 89.00 | Hurt |
| use gradient-based rather than random Gaussian perturbation (ablation) | Vaccine · default fine-tuning setting | harmful score: 56.00 -> 49.00 | Helped |
| use gradient-based rather than random Gaussian perturbation (ablation) | Vaccine · default fine-tuning setting | fine-tune accuracy: 73.80 -> 93.60 | Helped |
| use Double-LoRA instead of Single-LoRA (ablation) | Vaccine · SST2 | harmful score: 55.20 -> 48.90 | Helped |
| use Single-LoRA instead of Double-LoRA (ablation) | Vaccine · GSM8K | harmful score: 69.40 -> 68.40 | Helped |
| add EWC to user fine-tuning with lambda=1e12 (ablation) | Vaccine · SST2 | harmful score: 48.20 -> 43.00 | Helped |
| add EWC to user fine-tuning with lambda=1e12 (ablation) | Vaccine · SST2 | fine-tune accuracy: 93.80 -> 54.40 | Hurt |
Surprising findings
- Non-aligned model fine-tuned on purely benign data (SST2) showed lower harmful score than when harmful data is mixed in, and SFT-aligned models can have higher harmful score than the non-aligned model.
- Fine-tuning on benign data does not necessarily degrade safety, disputing a prior claim, depending on model and dataset.
Limitations
- Roughly 2x training overhead in the alignment stage relative to SFT.
- Weak extension to RLHF, as only SFT-based alignment is evaluated.
- Reduction in harmful score diminishes at high harmful ratios and Vaccine's fine-tune accuracy slightly drops at some settings; EWC outperforms it in harmful score for larger sample counts.
Experimental setup
Datasets: BeaverTails SST2 AGNEWS GSM8K AlpacaEval
Benchmarks: BeaverTails SST2 AGNEWS GSM8K AlpacaEval
Metrics: harmful score fine-tune accuracy alignment loss training time GPU memory
Baselines: Non-Aligned SFT EWC Vlguard KL
Builds on: FGSM SAM LoRA EWC
Closest papers by meaning
- Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbation (ICLR 2025 · similarity 0.90)
- Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuning Attack (ICML 2025 · similarity 0.88)
- Safety Alignment Should be Made More Than Just a Few Tokens Deep (ICLR 2025 · similarity 0.88)
- Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbation (NeurIPS 2025 · similarity 0.88)
- Differentially Private Steering for Large Language Model Alignment (ICLR 2025 · similarity 0.88)
- Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attack (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.