Research map / Label-efficient and robust learning / Knowledge distillation
Layer Attack Unlearning: Fast and Accurate Machine Unlearning via Layer Level Attack and Knowledge Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Layer Attack Unlearning (LAU), a class-wise unlearning method that runs a Partial-PGD attack only on the classification layer and then applies knowledge distillation with double softmax to unlearn a class quickly while preserving accuracy.
Problem
Class-wise machine unlearning: removing all samples of a requested class from a trained classifier quickly and accurately, without full retraining and without degrading accuracy on retained data.
Approach
Attack only the final fully connected layer (Partial-PGD on intermediate latent features) to find neighboring labels for forget data, then update only that layer with a combined cross-entropy and knowledge-distillation loss using a teacher copy and double softmax, inspired by Forward-Forward's layer-wise objectives.
Key results
- On CIFAR-10 with ResNet18, LAU reaches D_tr 93.53% and D_tf 0% with US 0.9504, versus 0.9428 for the Retrain (optimal) baseline and 0.8329 for Boundary Shrink.
- Partial-PGD unlearning takes 3.76 to 25.93 s on CIFAR-10 versus 14.18 to 237.36 s for original PGD with comparable or better accuracy.
- Across 12 model-dataset settings, the method reports an average US of 0.9443, outperforming the listed baselines in most cases.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Layer Attack Unlearning instead of Retrain | ResNet18 · CIFAR-10 | US: 0.9428 -> 0.9504 | Helped |
| replace Original PGD with Partial-PGD (ablation) | Layer Attack Unlearning on CIFAR-10, ResNet18 · CIFAR-10 | time (s): 18.19 -> 4.37 | Helped |
| replace Original PGD with Partial-PGD (ablation) | Layer Attack Unlearning on CIFAR-10, ResNet18 · CIFAR-10 | D_tr accuracy: 92.97 -> 93.53 | Helped |
| add Double Softmax (ablation) | Layer Attack Unlearning on Fashion-MNIST, VGG16 · Fashion-MNIST | D_tr accuracy: 84.74 -> 93.89 | Helped |
| add Double Softmax (ablation) | Layer Attack Unlearning on Fashion-MNIST, ResNet18 · Fashion-MNIST | D_tf accuracy: 0.1 -> 0 | Helped |
| reduce forget data to 50% (ablation) | Layer Attack Unlearning, CIFAR-10 VGG16 · CIFAR-10 | time: 3.76 -> 1.91 | Helped |
| reduce forget data to 10% (ablation) | Layer Attack Unlearning, CIFAR-10 VGG16 · CIFAR-10 | D_tf accuracy: 0 -> 0 | No clear effect |
| set KD alpha to 0.5 instead of 0 (ablation) | Layer Attack Unlearning, CIFAR-10 ResNet18 · CIFAR-10 | US: 0.9456 -> 0.9502 | Helped |
| set KD alpha to 1 instead of 0.5 (ablation) | Layer Attack Unlearning, CIFAR-10 ResNet18 · CIFAR-10 | US: 0.9502 -> 0.8849 | Hurt |
| set KD temperature to 4 instead of 1 (ablation) | Layer Attack Unlearning, CIFAR-10 ResNet18 · CIFAR-10 | D_tr accuracy: 93.4 -> 93.53 | Helped |
| increase KD temperature from 4 to 8 (ablation) | Layer Attack Unlearning, CIFAR-10 ResNet18 · CIFAR-10 | D_tr accuracy: 93.53 -> 93.32 | Hurt |
| set exponent of T to 2 instead of 1 (ablation) | Layer Attack Unlearning, CIFAR-10 ResNet18 · CIFAR-10 | US: 0.9493 -> 0.9504 | Helped |
Surprising findings
- Layer-only unlearning of the classification layer yields higher retain accuracy than unlearning the whole model in several settings.
- Reducing the forget set to 10% of its size retains similar accuracy and US on most CIFAR-10 models.
Limitations
- Evaluated only on image classification with small backbones (VGG16, ResNet18/50, ViT) and three datasets.
- Unlearning is limited to single-class removal in the main experiments; multi-class results were removed from the paper.
- The proposed US metric is a custom construction that is not standard and may not be directly comparable to other unlearning metrics.
Experimental setup
Datasets: CIFAR-10 Fashion-MNIST VGGFace2
Benchmarks: CIFAR-10 Fashion-MNIST VGGFace2
Metrics: accuracy on retain and forget sets test accuracy unlearning score (US) unlearning time (s)
Baselines: Retrain Negative Gradient Fine-tune Random Label Fisher Forgetting Boundary Shrink IWU
Builds on: PGD Forward-Forward Boundary Unlearning Knowledge Distillation
Closest papers by meaning
- Adversarial Machine Unlearning (ICLR 2025 · similarity 0.90)
- Efficient Verified Unlearning For Distillation (NeurIPS 2025 · similarity 0.89)
- Machine Unlearning Fails to Remove Data Poisoning Attacks (ICLR 2025 · similarity 0.89)
- Unlearned but Not Forgotten: Data Extraction after Exact Unlearning in LLM (NeurIPS 2025 · similarity 0.88)
- System-Aware Unlearning Algorithms: Use Lesser, Forget Faster (ICML 2025 · similarity 0.88)
- Certified Unlearning for Neural Networks (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.