Research map / Large language models / Language, safety and interpretability

SHRED: Retain-Set-Free Unlearning via Self-Distillation with Logit Demotion

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: LLM pruning and scaling data · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SHRED, a retain-set-free LLM unlearning method that demotes the memorized token logit only at the lowest-probability (high-surprisal) positions of forget-set sequences and uses a top-K KL self-distillation loss, with the remaining positions acting as implicit retain anchors.

Problem

LLM unlearning typically needs a curated retain set to preserve general utility, which creates data dependency and domain bias; retain-set-free alternatives often damage utility.

Approach

Select the bottom-P lowest-probability tokens in each forget sequence as forget positions, demote the memorized token's logit only there via modified KL targets, and keep the model's own distribution at the remaining (benign) positions, all trained with one top-K KL self-distillation objective and no retain data.

Key results

• On TOFU, SHRED reaches fkm 0.055 with MU 0.637 (Target: fkm 0.148, MU 0.612), with no retain set.
• On MUSE-Books, SHRED achieves fkm 0.237 and rkm 0.519, while GA and CEU show over-forgetting on that benchmark.
• In the relearning attack on TOFU, SHRED's forget KnowMem rises by +0.15 versus +0.47 for NPO and +0.67 for RMU.

Surprising findings

• Smaller batch sizes (down to BS=1) preserve utility better, the opposite of conventional supervised-learning wisdom.
• LoRA settles about 8 utility points below full-precision training, while 8-bit optimizers are essentially free.

Limitations

• Requires tuning of P (and pi for the nucleus variant), with no single setting guaranteed best.
• Only forgets concepts whose probes overlap with the forget corpus; WMDP-Cyber absolute reductions are small (about 3-5 points).
• LoRA degrades utility relative to full fine-tuning.

Experimental setup

Datasets: TOFU MUSE-News MUSE-Books RWKU Hubble WMDP-Cyber

Benchmarks: TOFU MUSE-News MUSE-Books RWKU Hubble WMDP-Cyber

Metrics: forget KnowMem forget VerbMem model utility PrivLeak relearning robustness WMDP accuracy

Baselines: GA GradDiff NPO SimNPO DPO RMU WHP TaskVec CEU RT UNDIAL

Builds on: UNDIAL NPO SimNPO RMU SDFT

Closest papers by meaning

• Constrained Entropic Unlearning: A Primal-Dual Framework for Large Language Models (NeurIPS 2025 · similarity 0.93)
• Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Difference (NeurIPS 2024 · similarity 0.92)
• What to Forget in Unlearning? Forget Set Curation for Language Models (NeurIPS 2026 · similarity 0.91)
• A Closer Look at Machine Unlearning for Large Language Models (ICLR 2025 · similarity 0.91)
• LLM Unlearning via Loss Adjustment with Only Forget Data (ICLR 2025 · similarity 0.91)
• Learning What to Forget: Improving LLM Unlearning via Learned Token-Level Importance (NeurIPS 2026 · similarity 0.91)