Research map / Large language models / Instruction and fine-tuning
Unified Parameter-Efficient Unlearning for LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LLMEraser, a PEFT-adapter unlearning framework that uses influence functions with a finite-sum quadratic-programming reformulation (mini-batch Adam on HVPs) to update LoRA adapters for instance removal, query modification, and response correction without retraining.
Problem
Unlearning instance-level data (removal, query fixes, response corrections) from LLMs fine-tuned with PEFT adapters such as LoRA, without costly retraining or model-architecture changes.
Approach
Apply influence-function parameter-change estimates directly to LoRA adapters for three instance-wise unlearning tasks, and compute the inverse Hessian-vector product by solving an equivalent finite-sum quadratic program with mini-batch Adam using Hessian-vector products, avoiding CG full-batch passes and stochastic-estimation error accumulation.
Key results
- On BookCrossing instance removal (5% removed), LLMEraser reaches AUC 0.6319 vs Retrain 0.6357, versus Gradient Ascent 0.6187 and E2URec 0.6205.
- On LastFM query modification with 10% noisy users, LLMEraser HitRatio@1 0.4463 vs Retrain 0.4508 and corrupted 0.4344 (Movielens 0.4456 vs Retrain 0.4565).
- Unlearning on QM task takes about 1.4e3 s vs 5.4e4 s for Retrain (about 31x speedup) on a single A100.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LLMEraser instead of the corrupted model to correct noisy user interactions | LLaRA · MovieLens, 10% interaction removal | HitRatio@1: 0.4222 -> 0.4456 | Helped |
| Use LLMEraser instead of the corrupted model to correct noisy user interactions | LLaRA · MovieLens, 5% interaction replacement | HitRatio@1: 0.4316 -> 0.4516 | Helped |
| Use LLMEraser for instance removal instead of retraining | TALLRec · BookCrossing, 5% instance removal | AUC: 0.6357 -> 0.6319 | No clear effect |
| Use LLMEraser to correct mislabeled responses instead of leaving the data corrupted | LLaVA 1.5-7B · MM-SPUBENCH, response correction | Average accuracy: 0.70 -> 0.81 | Helped |
| Use LLMEraser to correct mislabeled responses instead of leaving the data corrupted | LLaVA 1.5-7B · R-BENCH, response correction | F1-Score: 0.50 -> 0.63 | Helped |
| Use LLMEraser rather than retraining | LLaRA · QM, 5% interaction replacement | Execution time (s): $5.4\times10^{4}$ -> $1.4\times10^{3}$ | Helped |
| Increase LoRA rank from 8 to 32 (ablation) | LLMEraser on LLaRA · LastFM, 10% interaction replacement | HitRatio@1: 0.4426 -> 0.4180 | Hurt |
| Use LLMEraser instead of retraining for the adapter update | LLaRA · LastFM, 10% interaction replacement, LoRA r = 8 | GPU memory (MB): 33040 MB -> 30760 MB | Helped |
Surprising findings
- SISA and RecEraser fail to improve over the corrupted baseline on query modification, with average decreases of 7.0% and 31.3%.
Limitations
- Requires access to gradients of target data and the training set during unlearning.
- First-order Taylor approximation of the influence function introduces inherent estimation error.
- Experiments limited to a few LLM4Rec and MLLM settings with a single 7B backbone family.
Experimental setup
Datasets: BookCrossing MovieLens LastFM MM-SPUBENCH R-Bench
Benchmarks: BookCrossing MovieLens LastFM MM-SPUBENCH R-Bench
Metrics: AUC HitRatio@1 ValidRatio accuracy F1-score wall-clock execution time
Baselines: Retrain Corrupted SISA RecEraser Gradient Ascent E2URec APA
Builds on: influence functions LoRA SISA RecEraser E2URec
Closest papers by meaning
- Towards Robust and Parameter-Efficient Knowledge Unlearning for LLMs (ICLR 2025 · similarity 0.93)
- Invariance Makes LLM Unlearning Resilient Even to Unanticipated Downstream Fine-Tuning (ICML 2025 · similarity 0.92)
- In-Context Unlearning: Language Models as Few-Shot Unlearners (ICML 2024 · similarity 0.92)
- On Large Language Model Continual Unlearning (ICLR 2025 · similarity 0.92)
- Towards Effective Evaluations and Comparisons for LLM Unlearning Methods (ICLR 2025 · similarity 0.92)
- OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metrics (NeurIPS 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.