Research map / Continual learning / Continual fine-tuning of LLMs
Fast Machine Unlearning without Retraining through Selective Synaptic Dampening
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Selective Synaptic Dampening (SSD), a post hoc, retrain-free machine unlearning method that selects parameters via diagonal Fisher information and dampens them in proportion to their forget-set specialization.
Problem
Machine unlearning must forget specific training data while preserving performance on retained data, but state-of-the-art methods need retraining or stored training data, and existing retrain-free Fisher-based methods are very slow and underperform.
Approach
Two-step post hoc method: select parameters whose diagonal Fisher importance on the forget set exceeds alpha times their importance on the full training set, then dampen them by beta = min(lambda * I_D / I_Df, 1), so only the full-data Fisher needs to be stored rather than the training data.
Key results
- On CIFAR100 rocket class unlearning with ResNet18, SSD reduces MIA from a baseline of 93.40% to 2.20% (retrain 1.04%) while retain accuracy is 74.54% vs 72.83% for retrain.
- SSD runs in about 120 seconds versus roughly 5,871-28,744 seconds for Fisher Forgetting on ResNet18 tasks, i.e. orders of magnitude faster.
- Fisher Forgetting drops retain accuracy to 5.76% on CIFAR20 vehicle2 while SSD keeps it at 82.97%, versus 82.11% for retrain.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SSD instead of retraining | ResNet18 · CIFAR100 rocket class unlearning | execution time: 1804.93±9.76 -> 120.00±5.49 | Helped |
| Use SSD instead of Fisher Forgetting | ResNet18 · CIFAR20 Veh2 class unlearning | retain accuracy: 5.76±1.01 -> 82.97±0.00 | Helped |
| Use SSD instead of retraining | ResNet18 · CIFAR100 rocket class unlearning | retain accuracy: 72.83±0.42 -> 74.54±0.00 | Helped |
| Use SSD instead of retraining | Vision Transformer · CIFAR100 rocket class unlearning | MIA: 3.23±0.00 -> 5.40±0.00 | Hurt |
| Use SSD instead of retraining | ResNet18 · CIFAR20 rocket subclass unlearning | forget accuracy: 10.74±3.4 -> 2.17±0.00 | Helped |
| Use SSD instead of retraining | ResNet18 · CIFAR10 random observations unlearning | MIA: 74.22±0.04 -> 72.65±0.05 | No clear effect |
| Increase α from 10 to 20 (ablation) | SSD on ResNet18 · CIFAR100 rocket class unlearning | MIA: 2.20±0.00 -> 0.20 | Helped |
Surprising findings
- Amnesiac and Bad Teacher drive MIA near zero on CIFAR20 sea subclass unlearning, even though the retrained model retains a high MIA, suggesting a possible Streisand effect.
- Fisher Forgetting, the prior retrain-free Fisher method, fails to maintain retain-set accuracy despite its high computational cost.
Limitations
- SSD provides no certified or mathematical unlearning guarantee.
- Hyperparameters alpha and lambda must be chosen, and poor choices can degrade the model under repeated forget requests.
- Without a repair step, only a finite number of forget requests can be processed before retain performance degrades.
Experimental setup
Datasets: CIFAR10 CIFAR20 CIFAR100 PinsFaceRecognition
Benchmarks: CIFAR10 CIFAR20 CIFAR100 PinsFaceRecognition
Metrics: forget set accuracy retain set accuracy membership inference attack (MIA) execution time
Baselines: Retraining Finetune Bad Teacher Amnesiac UNSIR Fisher Forgetting
Builds on: Fisher Forgetting Bad Teacher UNSIR Amnesiac unlearning
Closest papers by meaning
- When to Forget? Complexity Trade-offs in Machine Unlearning (ICML 2025 · similarity 0.90)
- Efficient Source-free Unlearning via Energy-Guided Data Synthesis and Discrimination-Aware Multitask Optimization (ICML 2025 · similarity 0.89)
- SHRED: Retain-Set-Free Unlearning via Self-Distillation with Logit Demotion (NeurIPS 2026 · similarity 0.89)
- Forgetting Has Neighbors: Localized Collateral Forgetting in Machine Unlearning (NeurIPS 2026 · similarity 0.89)
- Learning What to Forget: Improving LLM Unlearning via Learned Token-Level Importance (NeurIPS 2026 · similarity 0.89)
- Selective Forgetting: Advancing Machine Unlearning Techniques and Evaluation in Language Models (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.