Research map / Large language models / Language, safety and interpretability

Learning What to Forget: Improving LLM Unlearning via Learned Token-Level Importance

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: LLM pruning and scaling data · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ATWU, which jointly learns token-level forget-specificity scores (a linear scorer over hidden states) and model parameters during LLM unlearning, using retain conflict as the selection criterion, and reports state-of-the-art forget-retain trade-offs on TOFU and RWKU.

Problem

Sample-level unlearning treats all tokens in a forget sample equally, which can damage general utility; existing token-weighted methods need external annotations, auxiliary models, or probability heuristics to decide which tokens are forget-specific.

Approach

Characterize forget-specific tokens as those whose forget loss can be minimized without conflicting with retain optimality; learn token weights via a linear scorer on hidden states jointly with the model by alternating updates on a budget-constrained, saturated negative-CE objective (SatGA+) with no external token labels.

Key results

• On TOFU forget10 with Llama-3.1-8B-Instruct, ATWU reaches UQ 91.7, exceeding the runner-up RMU (85.9) by 5.8 points.
• On RWKU ten-subject batch with Phi-3-Mini-4k-Instruct, ATWU reaches UQ 58.7, exceeding WGA (51.9) by 6.8 points.
• Learned token scores reach mean AUROC 75 ± 9 against ground-truth forget spans on TOFU, above the best baseline (~67-68).

Surprising findings

• Saturation alone (WGA, no token weighting) outperforms oracle ground-truth token labels paired with naive gradient ascent (UQ 41.5 vs 39.5).
• With adequate per-method tuning, RMU becomes highly competitive on TOFU, contradicting earlier literature.

Limitations

• Small forget sets provide weaker signal for the learned scorer, giving smaller gains on the smallest TOFU splits.
• Theory relies on a budget hyperparameter rho and a strict separation between forget-specific and structural tokens; the practical SatGA+ departs from the formal objective.
• Final evaluations are single-run and only two benchmarks and two model families are covered; judge is an LLM.

Experimental setup

Datasets: TOFU RWKU

Benchmarks: TOFU RWKU MMLU

Metrics: forget quality (FQ) retain degradation (RD) unlearning quality (UQ) MMLU repetitiveness win rate AUROC ES surrogate

Baselines: GradDiff DPO NPO SimNPO JensUn RMU WGA SatImp SEUL SU-LLM SU-Ngram FUNDIAL

Builds on: SatImp WGA GradDiff NPO SimNPO

Closest papers by meaning

• Selective Forgetting: Advancing Machine Unlearning Techniques and Evaluation in Language Models (AAAI 2025 · similarity 0.92)
• Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Difference (NeurIPS 2024 · similarity 0.92)
• RA$^2$: Retain-Anchored Attraction Preserves Forget-Adjacent Utility in LLM Unlearning (NeurIPS 2026 · similarity 0.92)
• What to Forget in Unlearning? Forget Set Curation for Language Models (NeurIPS 2026 · similarity 0.92)
• Forgetting Has Neighbors: Localized Collateral Forgetting in Machine Unlearning (NeurIPS 2026 · similarity 0.91)
• Keeping an Eye on LLM Unlearning: The Hidden Risk and Remedy (NeurIPS 2025 · similarity 0.91)