Research map / Continual learning / Continual fine-tuning of LLMs
Decoupled Distillation to Erase: A General Unlearning Method for Any Class-centric Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DELETE, a post-hoc, data-free (forget-set only) class unlearning method that decouples the unlearning loss into forgetting and retention terms and uses a masked distillation target from the frozen original model's logits.
Problem
Class-centric machine unlearning in pretrained DNNs when only the forget data is available (no remaining data, no pre-training intervention), where prior re-label methods over-forget or disturb the predictions of remaining classes.
Approach
Decompose the KL unlearning loss into forgetting and retention terms (inspired by decoupled KD); re-label methods implicitly enforce forgetting but give no retention supervision. DELETE sets the target class probability to zero via a mask on the frozen model's logits and renormalizes the rest, so the remaining classes keep their dark-knowledge relative magnitudes.
Key results
- On CIFAR-10 single-class forgetting, DELETE reaches 0% forget-test accuracy with 95.03% remain-test accuracy, versus 95.20% for the retrain model.
- On CIFAR-100 multi-class forgetting with 20 classes, DELETE keeps 1.20% forget-test accuracy and 79.30% remain-test accuracy, while competing methods fail to jointly achieve both.
- With a frozen linear classifier on CIFAR-10, DELETE achieves 0% Accft and 95.44% Accrt, outperforming all listed baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DELETE's masked distillation loss instead of the compared unlearning baselines | ResNet-18 · CIFAR-10, single-class forgetting | H-Mean: 95.96 -> 96.00 | Helped |
| Use DELETE's masked distillation loss instead of the compared unlearning baselines | ResNet-18 · CIFAR-100, single-class forgetting | H-Mean: 71.83 -> 71.47 | No clear effect |
| Use DELETE with a Swin-T backbone | Swin-T · Single-class forgetting | H-Mean: 82.73 -> 82.89 | Helped |
| Set α above the default to relax the forgetting condition (ablation) | DELETE · CIFAR-10, single-class forgetting | Accft: 0 -> 37.30; 0 -> 76.80; 0 -> 95.00 | Hurt |
| Set T above the default in the retention condition (ablation) | DELETE · CIFAR-10, single-class forgetting | Accrt: 95.03 -> 76.92; 95.03 -> 30.38; 95.03 -> 18.81 | Hurt |
Surprising findings
- Boundary Expand achieves strong performance in standard settings but drops sharply when the linear classifier is frozen, suggesting it unlearns mainly at the classifier level.
Limitations
- Evaluated mainly on small-scale image classification datasets (CIFAR, Tiny ImageNet) with ResNet-18 and similar architectures.
- Relies on a strict no-remaining-data setting; performance vs. methods using remaining data is shown only in grayed comparisons.
- Hyperparameter sensitivity: the shared learning rate and retention temperature are tuned and ablations show performance collapses with large alpha or temperature T.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny ImageNet VGGFace
Benchmarks: CIFAR-10 CIFAR-100 Tiny ImageNet VGGFace Pascal VOC
Metrics: forget accuracy remain accuracy forget test accuracy remain test accuracy H-Mean membership inference attack success rate
Baselines: Random Label Negative Gradient Fisher Forgetting Boundary Shrink Boundary Expand Learn to Unlearn Bad Teacher Saliency Unlearn SCRUB Influence Unlearn
Builds on: decoupled knowledge distillation Learn to Unlearn Boundary Shrink SCRUB Bad Teacher
Closest papers by meaning
- From Dormant to Deleted: Tamper-Resistant Unlearning Through Weight-Space Regularization (NeurIPS 2025 · similarity 0.88)
- Forgetting Has Neighbors: Localized Collateral Forgetting in Machine Unlearning (NeurIPS 2026 · similarity 0.88)
- Learning What to Forget: Improving LLM Unlearning via Learned Token-Level Importance (NeurIPS 2026 · similarity 0.88)
- Distribution-Level Feature Distancing for Machine Unlearning: Towards a Better Trade-off Between Model Utility and Forgetting (AAAI 2025 · similarity 0.88)
- Fast Machine Unlearning without Retraining through Selective Synaptic Dampening (AAAI 2024 · similarity 0.88)
- When to Forget? Complexity Trade-offs in Machine Unlearning (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.