Research map / Architectures and efficiency / Pruning and efficient tuning
LoRA Unlearns More and Retains More (Student Abstract)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PruneLoRA, a machine unlearning paradigm that prunes a model, adds LoRA adapters, and fine-tunes on the retain set to forget a class while preserving accuracy on remaining classes.
Problem
Approximate machine unlearning via sparsification or fine-tuning degrades performance on remaining classes and can be computationally costly, while exact retraining is too expensive.
Approach
Prune first, then adapt with LoRA low-rank updates, then unlearn by fine-tuning on the remaining data, so only a small set of adapter parameters is modified, reducing compute and memory and preserving retained-class performance.
Key results
- For ResNet-50, PruneLoRA achieved the highest remaining and testing accuracy among unlearning methods while keeping near-perfect unlearning accuracy.
- For ViT, PruneLoRA significantly outperformed other methods except LoRA on remaining and testing accuracy, and achieved perfect unlearning accuracy where LoRA had drastically low UA.
- The paper reports that PruneLoRA offers a balance of unlearning effectiveness, retained performance, and computational efficiency; exact numbers are in a table not included in the extracted text.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PruneLoRA instead of the other evaluated methods | ResNet-50 · CIFAR-10 unlearning; remaining and testing sets | remaining accuracy and testing accuracy | Helped |
| Use PruneLoRA instead of methods other than LoRA | ViT · CIFAR-10 unlearning; remaining and testing sets | remaining accuracy and testing accuracy | Helped |
| Combine pruning with LoRA rather than using LoRA alone | ViT · CIFAR-10 unlearning | unlearning accuracy | Helped |
| Restrict LoRA to the last attention layer instead of applying it to multiple attention layers (ablation) | ViT · CIFAR-10 unlearning | results | Helped |
Surprising findings
- Plain LoRA showed drastically low unlearning accuracy on ViT, whereas adding pruning before LoRA yielded perfect unlearning accuracy.
Limitations
- Experiments limited to CIFAR-10 with two architectures due to compute constraints.
- Evaluated only image classification; LLM and VLM applications not tested.
- Layer selection for LoRA was tuned empirically (e.g., only last attention layer for ViT).
Experimental setup
Datasets: CIFAR-10
Benchmarks: CIFAR-10
Metrics: Unlearning accuracy MIA-Efficacy Remaining accuracy Testing accuracy Run-time efficiency GPU memory
Baselines: Fine-tuning Pruning + Fine-tuning LoRA Retraining (exact unlearning)
Builds on: LoRA model sparsification unlearning
Closest papers by meaning
- When to Forget? Complexity Trade-offs in Machine Unlearning (ICML 2025 · similarity 0.89)
- In-Context Unlearning: Language Models as Few-Shot Unlearners (ICML 2024 · similarity 0.89)
- On Large Language Model Continual Unlearning (ICLR 2025 · similarity 0.89)
- Towards Scalable Exact Machine Unlearning Using Parameter-Efficient Fine-Tuning (ICLR 2025 · similarity 0.89)
- Machine Unlearning under Overparameterization (NeurIPS 2025 · similarity 0.89)
- What makes unlearning hard and what to do about it (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.