Research map / Large language models / Language, safety and interpretability
On Large Language Model Continual Unlearning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes O^3, an orthogonal-LoRA continual LLM unlearning framework with an OOD detector (contrastive entropy loss, glocal scoring) that softly decides whether to load the unlearning LoRA, without any retained data.
Problem
Continual unlearning requests for LLMs cause cumulative utility loss, and existing methods require retained data that is often unavailable due to privacy and copyright constraints.
Approach
Train one orthogonal-regularized LoRA per stream of unlearning requests and a separate unsupervised OOD detector (RoBERTa-large with contrastive entropy loss plus Mahalanobis/cosine glocal scoring and OCSVM) that produces a soft weight for loading the unlearning LoRA at inference, so retained data is never needed.
Key results
- O^3 achieves the highest U2R across the three tasks, with the paper stating it significantly surpasses the second-best method.
- On TOFU, O^3 reaches roughly 12-16% sample-level unlearning accuracy after the first two requests while keeping retained-distribution accuracy around 85%, close to base (85.8%).
- The OOD detector's full design reaches AUROC of 92.5-99.9 on TOFU and QA ID/OOD splits, outperforming MDF and Agg baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use O³ instead of the strongest reported TOFU baselines for the third request | LLaMA2-7B-chat continual unlearning · TOFU-forget10, request 3 | sample-level unlearning accuracy: 16.5_{\pm 0.3} -> 15.5_{\pm 0.7} | Helped |
| Use O³ instead of the strongest utility-preserving TOFU baseline | LLaMA2-7B-chat continual unlearning · TOFU-forget10, request 3 | retained distribution accuracy: 83.4_{\pm 0.1} -> 84.9_{\pm 0.2} | Helped |
| Use O³ instead of the strongest CLINC150 unlearning baseline | LLaMA2-7B continual unlearning · CLINC150 home, request 3 | sample-level unlearning accuracy: 0_{\pm 0} -> 40.6_{\pm 4.0} | Mixed |
| Replace contrastive entropy loss with SimCLR (ablation) | O³ OOD detector · TOFU-forget01, R.D. | AUROC: 97.8 -> 96.0 | Hurt |
| Replace contrastive entropy loss with MoCo (ablation) | O³ OOD detector · TOFU-forget01, R.D. | AUROC: 97.8 -> 95.8 | Hurt |
| Remove cosine distance from OOD scoring (ablation) | O³ OOD detector · TOFU-forget01, R.D. | AUROC: 97.8 -> 90.3 | Hurt |
| Remove Mahalanobis distance from OOD scoring (ablation) | O³ OOD detector · TOFU-forget01, R.D. | AUROC: 97.8 -> 97.0 | Mixed |
| Score using only the last layer instead of all layers (ablation) | O³ OOD detector · TOFU-forget01, R.D. | AUROC: 97.8 -> 91.3 | Hurt |
| Add orthogonal regularization (λ=0.1) rather than omit it (ablation) | O³ unlearning optimization · ScienceQA | sample-level unlearning accuracy: 15.3 -> 9.3 | Helped |
| Add orthogonal regularization (λ=0.1) rather than omit it (ablation) | O³ unlearning optimization · ScienceQA | retained distribution accuracy: 90.1 -> 91.1 | Helped |
| Replace soft-weighted inference with hard-weighted inference (ablation) | O³ inference · ScienceQA | sample-level unlearning accuracy: 9.3 -> 38.0 | Hurt |
| Increase soft-weight scaling from ζ=1 to ζ=100 (ablation) | O³ inference · ScienceQA | sample-level unlearning accuracy: 10.0 -> 9.1 | Helped |
Surprising findings
- Adding retained data from an unrelated distribution (CommonsenseQA) gives little benefit for preserving performance on the unlearning-susceptible distribution.
- Existing baselines such as GradDif and SOGD often produce empty or repetitive outputs, which the authors count as unlearning failures.
Limitations
- Relies on the unlearning data being available during unlearning and on stored score vectors for the OOD detector.
- Evaluated on three tasks with LLaMA2-7b only.
Experimental setup
Datasets: ScienceQA CommonsenseQA OpenbookQA TOFU CLINC150 MRPC RTE
Benchmarks: ScienceQA CommonsenseQA OpenbookQA TOFU CLINC150 MRPC RTE
Metrics: unlearning-utility ratio (U2R) accuracy on sample-level and distribution-level unlearning s retained and utility accuracy AUROC for OOD detection
Baselines: GradAsc GradDif EUL PO NPO SOGD SOPO
Builds on: LoRA MoCo TOFU SOPO Mahalanobis distance
Closest papers by meaning
- A Closer Look at Machine Unlearning for Large Language Models (ICLR 2025 · similarity 0.93)
- In-Context Unlearning: Language Models as Few-Shot Unlearners (ICML 2024 · similarity 0.92)
- OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metrics (NeurIPS 2025 · similarity 0.92)
- LLM Unlearning via Loss Adjustment with Only Forget Data (ICLR 2025 · similarity 0.92)
- Unified Parameter-Efficient Unlearning for LLMs (ICLR 2025 · similarity 0.92)
- Towards Effective Evaluations and Comparisons for LLM Unlearning Methods (ICLR 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.