Research map / Adversarial robustness and security / Attacks on text and graphs
Vulnerability-Aware Alignment: Mitigating Uneven Forgetting in Harmful Fine-Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Vulnerability-Aware Alignment (VAA), which partitions alignment data by how often examples are forgotten under a proxy harmful fine-tuning run and trains with an EXP3-based adversarial group sampler plus group-dependent weight perturbations (Group DRO) to balance learning across groups.
Problem
Harmful fine-tuning breaks safety alignment, and existing alignment-stage defenses treat all alignment examples equally, ignoring that some examples are much more prone to forgetting during fine-tuning.
Approach
Measure per-example vulnerability (ForgotNum) via a proxy fine-tuning run, split data into vulnerable and invulnerable groups, and apply group DRO with an adversarial EXP3 sampler and group-specific adversarial weight perturbations during alignment.
Key results
- On Llama2 7B across four fine-tuning tasks, VAA reaches average harmful score 24.82 vs 34.51 for SFT and 31.28 for Vaccine, with average FA 56.77.
- With varying harmful ratios, VAA averages harmful score 19.22 vs 31.61 for SFT while keeping FA around 90.9.
- On Qwen2.5-7B across five fine-tuning epochs, VAA gives harmful scores of 17.73 to 22.11 vs 26.89 to 33.27 for SFT, with comparable accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use vulnerability-aware alignment instead of Booster | Llama2 7B · SST2 | HS: 25.90 -> 20.00 | Helped |
| use vulnerability-aware alignment instead of RepNoise | Llama2 7B · AGNEWS | HS: 27.29 -> 21.12 | Helped |
| use vulnerability-aware alignment instead of Vaccine | Llama2 7B · GSM8K | HS: 34.66 -> 31.08 | Helped |
| use vulnerability-aware alignment instead of SFT | Llama2 7B · AlpacaEval | HS: 30.48 -> 27.09 | Helped |
| use vulnerability-aware alignment instead of Vaccine | Llama2 7B · Average across fine-tuning tasks | HS: 31.28 -> 24.82 | Helped |
| remove vulnerability-based example grouping (ablation) | VAA · Example grouping | HS: 20.00 -> 26.42 | Hurt |
| randomly swap 10% of examples between vulnerable and invulnerable groups (ablation) | VAA · Noisy example grouping | HS: 20.00 -> 21.08 | Hurt |
| sample exclusively from the vulnerable group (ablation) | VAA · Sampling strategies | HS: 20.00 -> 29.26 | Hurt |
| sample exclusively from the invulnerable group (ablation) | VAA · Sampling strategies | HS: 20.00 -> 33.98 | Hurt |
| replace dynamic adversarial sampling with importance sampling (ablation) | VAA · Sampling strategies | HS: 20.00 -> 28.64 | Hurt |
Surprising findings
- Sampling only from the invulnerable group yields worse harmful scores than sampling only from the vulnerable group, suggesting vulnerable examples carry more informative signal.
- Vulnerability groups estimated on Llama2 transferred to Qwen2.5 without re-clustering and still improved safety.
Limitations
- Data partitioning is simple and relies on a pseudo (proxy) fine-tuning run.
- Harmful score still rises with harmful data ratio and fine-tuning epochs; VAA does not prevent alignment breakdown entirely.
- Evaluation covers only Llama2 7B and Qwen2.5 7B and a limited set of fine-tuning tasks.
Experimental setup
Datasets: BeaverTails Alpaca SST-2 AG News GSM8K AlpacaEval
Benchmarks: SST-2 AG News GSM8K AlpacaEval BeaverTails test set
Metrics: Harmful Score (HS) Fine-tuning Accuracy (FA)
Baselines: SFT RepNoise Vaccine Booster
Builds on: Group DRO Vaccine Booster EXP3 RepNoise
Closest papers by meaning
- The Geometry of Alignment Collapse: When Fine-Tuning Breaks Safety (NeurIPS 2026 · similarity 0.89)
- Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templates (NeurIPS 2024 · similarity 0.88)
- Safety Alignment Should be Made More Than Just a Few Tokens Deep (ICLR 2025 · similarity 0.87)
- Learning and Forgetting Unsafe Examples in Large Language Models (ICML 2024 · similarity 0.87)
- Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (ICLR 2024 · similarity 0.87)
- Safety Alignment Can Be Not Superficial With Explicit Safety Signals (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.