Research map / Continual learning / Continual fine-tuning of LLMs
Toward Efficient Data-Free Unlearning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Inhibited Synthetic PostFilter (ISPF), which combines an inhibited generator loss and teacher-logit redistribution to improve retained-class accuracy in data-free unlearning via DFKD.
Problem
Data-free unlearning (removing forgetting classes without access to real or similar data) is inefficient: the existing GKT filters out synthetic samples likely to belong to the forgetting class, so over-filtering removes much retaining-related information and slows distillation.
Approach
Two complementary fixes: Inhibited Synthesis trains the generator with a loss that suppresses synthesis of forgetting-class samples; PostFilter keeps all synthetic samples and instead redistributes the teacher's forgetting-class logits to retaining classes so the student still receives retaining-related supervision.
Key results
- On SVHN with AllCNN, ISPF reaches A_r 92.68% vs GKT 55.23% with A_f 0% (Retrain A_r 94.14%).
- On CIFAR-10 with ResNet18, ISPF achieves A_r 83.33% vs GKT 58.11%, with MIA-II 29.36 vs Retrain 30.81.
- On CIFAR-100 multi-class unlearning (10 classes), ISPF achieves A_r 62.58% vs GKT 49.86%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace GKT with ISPF | Data-free unlearning with AllCNN · SVHN | retaining accuracy (A_r): 55.23$\pm$2.42 -> 92.68$\pm$0.13 | Helped |
| Replace GKT with ISPF | Data-free unlearning with AllCNN · CIFAR-10 | retaining accuracy (A_r): 57.62$\pm$1.13 -> 86.02$\pm$0.03 | Helped |
| Replace GKT with ISPF | Data-free unlearning with ResNet18 · CIFAR-100 multi-class | retaining accuracy (A_r): 49.86$\pm$3.01 -> 62.58$\pm$0.62 | Helped |
| Add Inhibited Synthesis to PostFilter (ablation) | Data-free unlearning with AllCNN · SVHN | retaining accuracy (A_r): 91.15$\pm$0.85 -> 92.68$\pm$0.13 | Helped |
| Add PostFilter to Inhibited Synthesis (ablation) | Data-free unlearning with AllCNN · CIFAR-10 | retaining accuracy (A_r): 84.93$\pm$0.02 -> 86.02$\pm$0.03 | Helped |
| Add Inhibited Synthesis to PostFilter (ablation) | Data-free unlearning with ResNet18 · CIFAR-10 | retaining accuracy (A_r): 81.21$\pm$0.72 -> 83.33$\pm$0.81 | Helped |
| Add PostFilter to Inhibited Synthesis (ablation) | Data-free unlearning with ResNet18 · SVHN | retaining accuracy (A_r): 90.75$\pm$0.55 -> 91.92$\pm$0.23 | Helped |
| Replace PostFilter with probability-distribution zeroing and renormalization (ablation) | ISPF with AllCNN · SVHN | retaining accuracy (A_r): 86.58$\pm$0.61 -> 92.68$\pm$0.13 | Helped |
| Replace PostFilter with probability-distribution zeroing and renormalization (ablation) | ISPF with AllCNN · CIFAR-10 | retaining accuracy (A_r): 76.16$\pm$0.21 -> 86.02$\pm$0.03 | Helped |
| Replace DFQ with ZSKT as the fundamental distillation method (ablation) | ISPF · SVHN | retaining accuracy (A_r): 67.33$\pm$2.93 | Mixed |
| Use ISPF instead of GKT (ablation) | ResNet18 data-free unlearning with ZSKT · SVHN | retaining accuracy (A_r): 25.49$\pm$5.25 -> 85.67$\pm$0.65 | Helped |
| Use a shared-superclass rather than distributed forgetting-class selection (ablation) | GKT · CIFAR-100 multi-class | retaining accuracy (A_r): 49.86$\pm$3.01 -> 56.04$\pm$1.02 | Helped |
Surprising findings
- Naively zeroing forgetting-class probabilities and renormalizing (PD) performs worse than the proposed logit redistribution, attributed to hard clipping of teacher outputs.
Limitations
- Forgetting-class accuracy relies on a PreFilter threshold hyperparameter delta tuned per dataset.
- Evaluated only on small image classification benchmarks (SVHN, CIFAR-10/100) with small CNN/ResNet architectures.
- Retaining accuracy still falls short of Retrain on several settings (e.g., CIFAR-10 ResNet18 83.33 vs 92.4).
Experimental setup
Datasets: SVHN CIFAR-10 CIFAR-100
Benchmarks: SVHN CIFAR-10 CIFAR-100
Metrics: forgetting accuracy A_f retaining accuracy A_r MIA-I MIA-II Anamnesis Index (AIN) wall time
Baselines: GKT BlockF DFKD Retrain PD
Builds on: Data-free Quantization (DFQ) Zero-shot Knowledge Transfer (ZSKT) Gated Knowledge Transfer (GKT) Knowledge distillation (KL divergence)
Closest papers by meaning
- Towards Source-Free Machine Unlearning (CVPR 2025 · similarity 0.89)
- Efficient Source-free Unlearning via Energy-Guided Data Synthesis and Discrimination-Aware Multitask Optimization (ICML 2025 · similarity 0.87)
- When to Forget? Complexity Trade-offs in Machine Unlearning (ICML 2025 · similarity 0.87)
- The Utility and Complexity of In- and Out-of-Distribution Machine Unlearning (ICLR 2025 · similarity 0.87)
- Certified Unlearning for Neural Networks (ICML 2025 · similarity 0.87)
- SHRED: Retain-Set-Free Unlearning via Self-Distillation with Logit Demotion (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.