Research map / Label-efficient and robust learning / Knowledge distillation
Efficient Verified Unlearning For Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PURGE, a SISA-based framework for knowledge distillation that maps teacher constituent models to student constituent models and uses incremental multi-teacher soft labels so teacher-side unlearning only partially retrains affected students.
Problem
Verified (exact) machine unlearning in teacher-student knowledge distillation: naive SISA on both ensembles forces full student retraining whenever a teacher's data is removed.
Approach
Constituent mapping assigns each teacher constituent to exactly one student constituent's teacher ensemble, and an incremental multi-teacher schedule over chunks confines each teacher's influence to later chunks, so teacher unlearning only requires partial retraining of the affected student models while preserving SISA-style checkpointing.
Key results
- With M=32 teachers and N=32 students, PURGE retrains in 23.17 s per teacher unlearning request versus 737.14 s for SISA (about 32x speed-up).
- On MNIST and SVHN with M=N=32, PURGE reaches 97.16% and 83.09% accuracy versus 97.08% and 83.44% for SISA.
- On ARC-challenge with Qwen2.5-32B teacher and Qwen2.5-3B students, PURGE achieves 26.09% vs 26.42% for naive SISA, with the teacher at 27.76%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use constituent mapping and incremental multi-teacher distillation instead of naive SISA distillation | student retraining after teacher-side unlearning on MNIST · MNIST | student retraining time per request: 737.14±10.08 seconds -> 23.17 ± 0.17 seconds; ≈ 32× speed-up | Helped |
| use PURGE instead of naive SISA | student updates for teacher-side unlearning on ARC-Challenge · ARC-Challenge | student update time: 18.9 GPU-hours -> 2.36 GPU-hours | Helped |
| use PURGE instead of SISA | student distillation with M=32 and N=32 · MNIST | accuracy: 97.08% -> 97.16% | No clear effect |
| use PURGE instead of SISA | student distillation with M=32 and N=32 · SVHN | accuracy: 83.44% -> 83.09% | No clear effect |
| use incremental multi-teacher soft labels instead of single-teacher soft labels (ablation) | student distillation with M=32 and N=1 · MNIST | accuracy drop relative to SISA: 0.14% on MNIST | Helped |
| use incremental multi-teacher soft labels instead of single-teacher soft labels (ablation) | student distillation with M=32 and N=1 · SVHN | accuracy drop relative to SISA: 0.18% on SVHN; 7.35% on SVHN | Helped |
| replace incremental multi-teacher labels with single-teacher soft labels (ablation) | student distillation when N is small · MNIST and SVHN | accuracy: 1.30% on MNIST; 7.35% on SVHN | Hurt |
| increase the number of student constituent models N | PURGE teacher-side unlearning · MNIST | student retraining speed-up: ≈ 32× speed-up | Helped |
| increase slices per chunk from r=1 to r=4 | PURGE teacher-side unlearning · MNIST | student retraining speed-up | Hurt |
| use PURGE instead of naive SISA | student distillation on ARC-Challenge · ARC-Challenge | accuracy: 26.42% -> 26.09% | Hurt |
| use matched label-imbalanced teacher and student shards instead of single-teacher labels (ablation) | student distillation with 1 student · SST5 | accuracy: $45.02\%$ -> $50.59\%$ | Helped |
Surprising findings
- Single-teacher soft labels cause large accuracy drops (e.g., 7.35% on SVHN with N=1), while the incremental multi-teacher schedule stays close to SISA.
- On the ARC reasoning task both PURGE and naive SISA perform near chance, with the teacher only at 27.76%.
Limitations
- Storage overhead from storing intermediate checkpoints for every slice (e.g., 386GB student checkpoints).
- Accuracy degrades with more student constituents due to less data per model; ARC reasoning results are near chance.
- Assumes an honest service provider for verified unlearning guarantees; multi-teacher aggregation is simple averaging.
Experimental setup
Datasets: MNIST SVHN CIFAR-100 SST5 ARC
Benchmarks: MNIST SVHN CIFAR-100 SST5 ARC-challenge
Metrics: accuracy unlearning retraining time speed-up over SISA
Baselines: SISA Single-teacher soft label ablation Teacher ensemble SCRUB RKLD
Builds on: SISA knowledge distillation LoRA
Closest papers by meaning
- Layer Attack Unlearning: Fast and Accurate Machine Unlearning via Layer Level Attack and Knowledge Distillation (AAAI 2024 · similarity 0.89)
- A Certified Unlearning Approach without Access to Source Data (ICML 2025 · similarity 0.89)
- Efficient Source-free Unlearning via Energy-Guided Data Synthesis and Discrimination-Aware Multitask Optimization (ICML 2025 · similarity 0.89)
- Certified Unlearning for Neural Networks (ICML 2025 · similarity 0.88)
- The Utility and Complexity of In- and Out-of-Distribution Machine Unlearning (ICLR 2025 · similarity 0.88)
- Unlearning through Knowledge Overwriting: Reversible Federated Unlearning via Selective Sparse Adapter (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.