Research map / Continual learning / Continual fine-tuning of LLMs
Adaptive Localization of Knowledge Negation for Continual LLM Unlearning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ALKN, a task-vector-based continual LLM unlearning method using entropic soft labels, learnable dynamic gradient masks, and adaptive learning rates, evaluated on TOFU, MUSE News, and WHP with a new TRAVIS utility corpus.
Problem
Continual unlearning in LLMs causes accumulating and amplified utility degradation as successive unlearning requests arrive, which existing single-shot unlearning methods handle poorly.
Approach
Build continual unlearning on task vectors and add three modules to the fine-tuning step: soft labels derived from the current model's predictions to avoid over-unlearning, a learnable dynamic gradient mask that encourages different tasks to update distinct parameters, and a reduced learning rate for parameters important to earlier tasks.
Key results
- On TOFU with Phi, ALKN keeps model utility at 0.4534 after five tasks versus 0.0812 for GA+RT and 0.1841 for NPO+RT, with F-Rouge 0.3314.
- On WHP (Llama-2-7B), ALKN reaches T-Rouge 0.5617 and R-Rouge 0.4586, above the best baseline WAGLE (0.4594 T-Rouge, 0.3721 R-Rouge), with F-Rouge 0.1790.
- The authors report that after sufficient unlearning ALKN retains over 95% of its model utility while some baselines lose utility completely.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ALKN instead of the strongest listed baseline | Phi · TOFU Task 5 | MU: 0.3938 -> 0.4534 | Helped |
| Use ALKN instead of SO-NPO | Llama-2-7B · WHP, final after three tasks | R-Rouge: 0.4109 -> 0.4586 | Helped |
| Use ALKN instead of SO-NPO | Llama-2-7B · WHP, final after three tasks | T-Rouge: 0.4536 -> 0.5617 | Helped |
| add entropic soft labels (ablation) | ALKN · TOFU Task 5 | MU: 0.3351 -> 0.4534 | Helped |
| add dynamic gradient sparsity (ablation) | ALKN · TOFU Task 5 | MU: 0.3893 -> 0.4534 | Helped |
| add adaptive parameter modulation (ablation) | ALKN · TOFU Task 5 | MU: 0.3768 -> 0.4534 | Helped |
Surprising findings
- Standard GA+RT and NPO+RT show cascading over-unlearning in later tasks, with utility dropping below 0.2 on TOFU.
- Under comparable unlearning levels, continual unlearning causes larger utility loss than independent unlearning of each task on TOFU.
Limitations
- Evaluated only on Llama-2-7B and Phi-1.5 with a limited set of datasets.
- Relies on TRAVIS, a synthetic utility corpus generated by the models themselves, whose sensitivity was only checked with Gaussian noise.
Experimental setup
Datasets: TOFU MUSE News WHP TRAVIS
Benchmarks: TOFU MUSE News WHP TRAVIS
Metrics: F-Rouge F-Prob MIA Min-K% Prob Forget Quality R-Rouge R-Prob T-Rouge Model Utility
Baselines: GA+RT GA+KL NPO+RT NPO+KL DPO+RT Task Vectors SKU EUL SO-NPO WAGLE
Builds on: task vectors gradient ascent unlearning NPO WAGLE min-k% prob membership inference
Closest papers by meaning
- In-Context Unlearning: Language Models as Few-Shot Unlearners (ICML 2024 · similarity 0.91)
- On Large Language Model Continual Unlearning (ICLR 2025 · similarity 0.91)
- Wisdom is Knowing What not to Say: Hallucination-Free LLMs Unlearning via Attention Shifting (NeurIPS 2025 · similarity 0.91)
- Towards Robust and Parameter-Efficient Knowledge Unlearning for LLMs (ICLR 2025 · similarity 0.91)
- Unified Parameter-Efficient Unlearning for LLMs (ICLR 2025 · similarity 0.91)
- Invariance Makes LLM Unlearning Resilient Even to Unanticipated Downstream Fine-Tuning (ICML 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.