Research map / Continual learning / Continual fine-tuning of LLMs
ESC: Erasing Space Concept for Knowledge Deletion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Knowledge Deletion (KD), a machine unlearning setting that removes feature-level knowledge using only forgetting data, with a training-free SVD-based method (ESC) and a learnable-mask variant (ESC-T), evaluated by a new Knowledge Retention score.
Problem
Existing machine unlearning methods mainly modify the classification head and leave substantial class knowledge in the feature extractor, so forgotten knowledge can still be recovered via linear probing or leaked; they also typically require the full remaining training data.
Approach
Compute principal directions of forgetting-data features via SVD and project features onto the subspace excluding the top principal directions (ESC), optionally refining the directions with a learnable mask trained by a penalized cross-entropy loss that only uses forgetting data (ESC-T).
Key results
- ESC-T reaches 0.00% forgetting accuracy on CIFAR-10 with harmonic mean 92.51 on retained test data, close to Retrain (92.48), while using only forgetting data.
- ESC is 9.17 times faster than the previously fastest method LAU across settings.
- On CIFAR-100-KR, ESC-T achieves HM 96.47 versus 79.43-ish for prior forgetting-only baselines such as NG and RL, showing feature-level erasure.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ESC-T instead of ESC | ESC on CIFAR-10 with All-CNN · CIFAR-10 | HM: 93.43 ± 0.03 -> 98.60 ± 0.10 | Helped |
| use ESC-T instead of ESC | ESC on CIFAR-100 with ResNet-18 · CIFAR-100 | HM: 94.08 ± 0.17 -> 95.40 ± 0.30 | Helped |
| use ESC-T instead of ESC | ESC on Tiny-ImageNet with ViT · Tiny-ImageNet | HM: 98.10 ± 0.02 -> 98.21 ± 0.12 | Helped |
| add mask learning with penalized cross-entropy loss (ablation) | ESC on CIFAR-100 · CIFAR-100 | HM: 99.01 -> 99.05 | No clear effect |
| add clipping to restrict mask values to 0–1 (ablation) | ESC-T on CIFAR-100-KR · CIFAR-100-KR | HM: 92.03 -> 94.70 | Helped |
| apply thresholding to the learned mask (ablation) | ESC-T on CIFAR-100 · CIFAR-100 | HM: 99.04 -> 90.33 | Hurt |
| use mask learning, clipping, and thresholding together (ablation) | ESC-T on CIFAR-100 · CIFAR-100 | HM: 90.33 -> 99.05 | Helped |
| increase pruning hyperparameter p from 1.7% to 50% (ablation) | ESC on CIFAR-100 · CIFAR-100 | HM: 99.01 -> 95.75 | Hurt |
| increase pruning hyperparameter p from 1.7% to 50% (ablation) | ESC on CIFAR-100-KR · CIFAR-100-KR | HM: 98.30 -> 76.66 | Hurt |
| vary threshold τ across 0.6, 0.7, 0.8, and 0.9 (ablation) | ESC-T on CIFAR-100 · CIFAR-100 | HM: 98.60; 99.05; 99.06; 98.26 | No clear effect |
Surprising findings
- Existing MU methods retain substantial feature-level knowledge: retraining only a new head on a frozen unlearned extractor recovers much accuracy on forgotten classes.
- Pruning-based l1-sparse performs comparably among MU baselines, yet features remain highly entangled across classes in the original model.
Limitations
- Relies on a pruning hyperparameter p and threshold tau chosen empirically (tau 0.7-0.8).
- Evaluated mainly on small-scale image classification (CIFAR, Tiny-ImageNet, Lacuna-10) with All-CNN, ResNet-18 and ViT.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny-ImageNet Lacuna-10
Benchmarks: CIFAR-10 CIFAR-100 Tiny-ImageNet Lacuna-10
Metrics: accuracy on forgetting and remaining data Knowledge Retention score (KR) harmonic mean MIA running time average gap
Baselines: Finetune Fisher SCRUB l1-sparse SalUn NG RL BS LAU BadT
Builds on: SVD-based unlearning (Kodge et al.) Random Label unlearning Negative Gradient unlearning LAU Gu et al. principal direction deactivation
Closest papers by meaning
- Decoupled Distillation to Erase: A General Unlearning Method for Any Class-centric Tasks (CVPR 2025 · similarity 0.86)
- On Local Overfitting and Forgetting in Deep Neural Networks (AAAI 2025 · similarity 0.85)
- Distribution-Level Feature Distancing for Machine Unlearning: Towards a Better Trade-off Between Model Utility and Forgetting (AAAI 2025 · similarity 0.85)
- From Dormant to Deleted: Tamper-Resistant Unlearning Through Weight-Space Regularization (NeurIPS 2025 · similarity 0.85)
- Label-Agnostic Forgetting: A Supervision-Free Unlearning in Deep Models (ICLR 2024 · similarity 0.85)
- The Unseen Threat: Residual Knowledge in Machine Unlearning under Perturbed Samples (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.