Research map / Large language models / Language, safety and interpretability

RA$^2$: Retain-Anchored Attraction Preserves Forget-Adjacent Utility in LLM Unlearning

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Pruning and circuits in LLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RA2, a latent-space unlearning method that attracts forget-token hidden states toward nearby retain-derived anchors at a reshape layer while keeping higher-layer retain states close to the frozen model, improving forget-adjacent utility on WMDP and MUSE.

Problem

LLM unlearning is usually judged by target suppression and clean retain utility, which miss degradation on benign prompts that lie near the forgotten domain (forget-adjacent utility loss).

Approach

Instead of only pushing forget-token representations away from the harmful computation, RA2 chooses a destination: each forget state is attracted (cosine loss) toward a local convex neighborhood of retain-derived anchors at the reshape layer, with a retain-consistency loss at a higher layer to the frozen original model.

Key results

• On Zephyr-7B-β, RA2 reaches 56.6 MMLU with WMDP-Bio 29.8 and 42.2 on WMDP→MMLU versus 29.5 for RMU and 24.5 for RMU-LAT, with forget-adjacent A− of 56.7 (+6.8% vs original).
• On MUSE-Books, RA2 gives the lowest verbatim memorization (13.13) among unlearning methods, versus 17.66 for RMU and 18.09 for RMU-LAT.
• Repel-direction and no-attraction ablations show that attraction direction, not only strength, governs forget-adjacent utility (WMDP→MMLU 24.0 for repel vs 43.0 default).

Surprising findings

• Repulsive forget-side updates achieve stronger direct forgetting on WMDP but sharply degrade forget-adjacent utility, so better forgetting does not imply better benign behavior near the forget boundary.
• Removing the retain-consistency term collapses MMLU to 26.8 despite strong forgetting, showing lower-layer redirection alone is insufficient.

Limitations

• Forget-adjacent utility is evaluated on a limited set of constructed perturbation settings (MMLU Perturbed, WMDP→MMLU, Bio-MNCP) that may not cover all real forget-adjacent prompts.
• RA2 adds anchor-field storage and nearest-neighbor search overhead, and is modestly slower than RMU in wall-clock time.
• Some generated outputs show repetition and truncation, and the anchor count appears to be an empirical choice rather than universal.

Experimental setup

Datasets: WikiText WMDP bio/cyber forget corpora MUSE-Books

Benchmarks: WMDP MMLU TruthfulQA MMLU Perturbed WMDP→MMLU Bio-MNCP MUSE-Books

Metrics: accuracy log-likelihood ROUGE-L verbatim memorization knowledge memorization privacy leak retain knowledge

Baselines: GA GradDiff JensUn RMU RMU-LAT NPO Task Vector WHP

Builds on: RMU RMU-LAT NPO WMDP MUSE

Closest papers by meaning

• Learning What to Forget: Improving LLM Unlearning via Learned Token-Level Importance (NeurIPS 2026 · similarity 0.92)
• RULE: Reinforcement UnLEarning Achieves Forget-retain Pareto Optimality (NeurIPS 2025 · similarity 0.91)
• Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Difference (NeurIPS 2024 · similarity 0.91)
• Keeping an Eye on LLM Unlearning: The Hidden Risk and Remedy (NeurIPS 2025 · similarity 0.90)
• What to Forget in Unlearning? Forget Set Curation for Language Models (NeurIPS 2026 · similarity 0.90)
• LLM Unlearning via Loss Adjustment with Only Forget Data (ICLR 2025 · similarity 0.90)