Research map / Architectures and efficiency / Pruning and efficient tuning
Towards Scalable Exact Machine Unlearning Using Parameter-Efficient Fine-Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes S3T, an exact unlearning framework that trains LoRA layers sequentially on disjoint data slices across multiple slice permutations so deletions are handled by deactivating layers rather than retraining.
Problem
Exact machine unlearning (SISA-style sharded, sliced training) requires retraining affected components, which causes production downtime and service failure, and existing methods fail quickly when early slices are deleted.
Approach
Sequential slice-wise LoRA training where the i-th layer is trained only on slices 1..i, so deletions are handled by deactivating LoRA layers; multiple models are trained on diverse slice permutations chosen by cyclic rotation or bipartite matching (BMS) to increase the deletion rate.
Key results
- On instruction tuning with Alpaca, S3T on Llama3-8B reaches comparable zero-shot performance to full fine-tuning (e.g., MMLU 59.6 vs 59.2 FT; PIQA 82.4 vs 80.9), and matches or beats FT on many tasks.
- On CIFAR-10/100 with m=5 shards and L=4 slices, S3T with budget B=4 approaches the performance of B=24 and handles more deletion requests than SISA and ProtoSISA.
- S3T achieves up to 2.3x and 3x deletion-rate gains over SISA for increasing shards and slices respectively; after 1000 deletion requests on CIFAR10 it reduces total deletion time 2.8x vs SISA and 25x vs full retraining.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use slice-wise S³T instead of full fine-tuning | Llama2-7B instruction tuning · MMLU, HellaSwag, PIQA, Winogrande, ARC-c, ARC-e, TruthfulQA, OBQA | accuracy: 43.3 -> 43.6; 77.4 -> 77.7; 79.9 -> 79.7; 69.6 -> 70.5; 50.4 -> 51.4; 74.7 -> 74.1; 46.3 -> 43.0; 46.8 -> 46.4 | Mixed |
| Use slice-wise S³T instead of full fine-tuning | Llama2-13B instruction tuning · MMLU, HellaSwag, PIQA, Winogrande, ARC-c, ARC-e, TruthfulQA, OBQA | accuracy: 51.8 -> 51.6; 81.3 -> 80.9; 82.0 -> 81.8; 72.2 -> 72.8; 55.0 -> 54.5; 79.3 -> 80.6; 38.8 -> 39.6; 46.6 -> 46.2 | Mixed |
| Use slice-wise S³T instead of full fine-tuning | Llama3-8B instruction tuning · MMLU, HellaSwag, PIQA, Winogrande, ARC-c, ARC-e, TruthfulQA, OBQA | accuracy: 59.2 -> 59.6; 80.8 -> 80.2; 80.9 -> 82.4; 72.2 -> 73.8; 59.0 -> 57.0; 79.6 -> 80.5; 46.6 -> 48.5; 46.8 -> 47.0 | Mixed |
| Increase S³T budget from B=8 relative to SISA | S³T · Deletion requests handled (m=5 shards, L=32 slices) | deletion rate: 1.6x | Helped |
| Use S³T instead of SISA with the same number of shards or slices | Exact unlearning deletion · Deletion rate | deletion rate: up to 2.3x; 3x | Helped |
| Use S³T instead of SISA or full retraining | CIFAR-10 unlearning · After 1000 deletion requests | total deletion time: 2.8x; 25x | Helped |
| Select slice sequences with cyclic rotation instead of random sampling | S³T sequence selection · Uniform deletion prior, budgets 1 ≤ B ≤ 120, L=5 | average edit distance | Helped |
| Select slice sequences with BMS instead of random sampling | S³T sequence selection · Non-uniform deletion priors | sequence score | Helped |
| Select slice sequences with BMS instead of random sampling | S³T sequence selection · Non-uniform deletion priors | average edit distance: L | Helped |
| Increase the number of training sequences beyond L | S³T unlearning system · Deletion rate | deletion rate | No clear effect |
| Increase the number of training sequences | S³T unlearning system · Performance retention after deletion requests | probability of retaining at least F(k) | Helped |
| Increase the number of slices affected by deletion (ablation) | S³T vision models · Vision datasets | relative performance drop: <5% | Hurt |
Surprising findings
- Slice-wise LoRA training sometimes outperforms full fine-tuning (e.g., 2.5% accuracy gain on Tiny ImageNet with ViT-Large).
- Deletion rate does not improve when the budget exceeds the number of slices L for the uniform prior, per the theoretical analysis.
Limitations
- Performance degrades when many layers are switched off, and increasing slice count can cause significant performance drop; a trade-off between slices and layers per slice is needed.
- Training budget multiplies training cost with number of sequences; BMS is only shown to yield at most L sequences, with conditional sampling for B>L.
- Theoretical guarantee for cyclic rotation sequences is not proven; the paper leaves it to future work.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny ImageNet GLUE SuperGLUE Alpaca
Benchmarks: CIFAR-10 CIFAR-100 Tiny ImageNet GLUE SuperGLUE MMLU HellaSwag PIQA Winogrande ARC TruthfulQA OpenBookQA
Metrics: accuracy Matthews correlation Pearson correlation deletion rate edit distance deletion time
Baselines: SISA ProtoSISA full re-training full fine-tuning (FT)
Builds on: SISA LoRA ProtoSISA
Closest papers by meaning
- Fast Exact Unlearning for In-Context Learning Data for LLMs (ICML 2025 · similarity 0.90)
- When to Forget? Complexity Trade-offs in Machine Unlearning (ICML 2025 · similarity 0.89)
- LoRA Unlearns More and Retains More (Student Abstract) (AAAI 2025 · similarity 0.89)
- What makes unlearning hard and what to do about it (NeurIPS 2024 · similarity 0.89)
- LoTUS: Large-Scale Machine Unlearning with a Taste of Uncertainty (CVPR 2025 · similarity 0.89)
- System-Aware Unlearning Algorithms: Use Lesser, Forget Faster (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.