Research map / Adversarial robustness and security / Attacks on text and graphs
Towards LLM Unlearning Resilient to Relearning Attacks: A Sharpness-Aware Minimization Perspective and Beyond
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that sharpness-aware minimization (SAM) and other smoothness optimizers (RS, GP, CR, WA) make LLM unlearning more resistant to relearning attacks and jailbreaks, evaluated on WMDP and MUSE.
Problem
LLM unlearning is fragile: a small number of forget-set samples can relearn removed knowledge (relearning attacks), and input jailbreaks can recover it.
Approach
Frames robust unlearning as a min-max problem with a relearning adversary that perturbs weights, which reduces to SAM applied to the forget loss; then explores other smoothness techniques (randomized smoothing, gradient penalty, curvature regularization, weight averaging) as analogous defenses.
Key results
- Under Relearn20 on WMDP, NPO+SAM reaches UE 0.70 versus 0.57 for NPO, with UT about 0.42 vs 0.44.
- On LLaMA-3 8B with 60 relearning samples, NPO+SAM achieves UE 0.70 under attack (matching TAR at 0.70) at 11.5 min versus 7441.9 min for TAR.
- NPO+SAM and NPO+RS retain UE under jailbreaking attacks where NPO suffers a significant drop.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SAM to NPO | NPO · WMDP, 20 forget samples | UE: 0.57 -> 0.70 | Helped |
| Add SAM to NPO | NPO · WMDP, 60 forget samples | UE: 0.37 -> 0.45 | Helped |
| Add SAM to NPO | NPO · WMDP, no attack | UE: 0.74 -> 0.74 | No clear effect |
| Add randomized smoothing, curvature regularization, gradient penalty, or weight averaging to NPO | NPO · WMDP, 20 forget samples | UE: NPO 0.57; NPO+RS 0.65; NPO+CR 0.62; NPO+GP 0.61; NPO+WA 0.69 | Helped |
| Add SAM to NPO | NPO · WMDP, 60 samples/1 epoch | UE under attack: 0.41 -> 0.70 | Helped |
| Add SAM to NPO | NPO · MUSE News, relearning attack | VerbMem on forget set: 56.57 -> 51.47 | Helped |
| Add SAM to NPO | NPO · MUSE Books, relearning attack | VerbMem on forget set: 67.52 -> 58.38 | Helped |
| Set SAM perturbation radius to 0.001 instead of 0.01 (ablation) | NPO+SAM · MUSE Books, relearning attack | VerbMem on forget set: 58.38 -> 70.9 | Hurt |
| Set SAM perturbation radius to 0.1 instead of 0.01 (ablation) | NPO+SAM · MUSE Books | unlearning effectiveness and relearning robustness: VerbMem: 58.38 -> 52.96; KnowMem: 38.33 -> 40.52 | Mixed |
| Increase the number of parameters perturbed by SAM in RMU from layers 5–7 to layers 1–7 (ablation) | RMU+SAM · WMDP, 20 samples/3 epochs | relearning robustness: 2.43%; 5.68% | Helped |
| Add SAM to GradDiff | GradDiff · WMDP relearning attacks | UE: 0.45 -> 0.65 (N = 20); 0.36 -> 0.44 (N = 60) | Helped |
| Add SAM to RMU | RMU · WMDP relearning attacks | UE: 0.39 -> 0.42 (N = 20); 0.36 -> 0.40 (N = 60) | Helped |
Surprising findings
- Smoothness-based variants also improve robustness to input-level jailbreaking attacks, which the authors attribute to 'shallow unlearning alignment' in the first few output tokens.
Limitations
- Relies on heuristic first-order SAM approximation rather than exact second-order analysis.
- Large rho degrades unlearning effectiveness, so the perturbation radius requires tuning.
- Evaluated on WMDP Bio and MUSE only with a limited set of models.
Experimental setup
Datasets: WMDP MUSE Wikitext AGNews GSM8K SST2 MMLU
Benchmarks: WMDP MUSE MMLU
Metrics: unlearning effectiveness (1-accuracy on WMDP Bio) utility (MMLU accuracy) KnowMem VerbMem runtime
Baselines: NPO GradDiff RMU RMU-LAT TAR
Builds on: negative preference optimization sharpness-aware minimization representation misdirection for unlearning adversarial training tampering attack resistance
Closest papers by meaning
- Towards Robust Knowledge Unlearning: An Adversarial Framework for Assessing and Improving Unlearning Robustness in Large Language Models (AAAI 2025 · similarity 0.91)
- From Dormant to Deleted: Tamper-Resistant Unlearning Through Weight-Space Regularization (NeurIPS 2025 · similarity 0.91)
- Keeping an Eye on LLM Unlearning: The Hidden Risk and Remedy (NeurIPS 2025 · similarity 0.91)
- Adversarial Machine Unlearning (ICLR 2025 · similarity 0.91)
- Unlearning or Obfuscating? Jogging the Memory of Unlearned LLMs via Benign Relearning (ICLR 2025 · similarity 0.90)
- Machine Unlearning Fails to Remove Data Poisoning Attacks (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.