Research map / Large language models / Language, safety and interpretability

Catastrophic Failure of LLM Unlearning via Quantization

ICLR 2025 · Large language models › Language, safety and interpretability · approach: LLM unlearning and benchmarks · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that 4-bit quantization (RTN, GPTQ, AWQ) of LLMs after unlearning restores much forgotten knowledge on MUSE (NEWS, BOOKS) and RWKU, explains it via weight-interval analysis, and proposes SURE (saliency-based, large-learning-rate unlearning) to resist quantization recovery.

Problem

Existing LLM unlearning methods with utility regularization may only hide knowledge; applying post-training quantization to unlearned models can recover forgotten information, which current benchmarks fail to detect.

Approach

Minimal weight changes from small learning rates plus utility regularization leave unlearned weights within the same quantization bins as the target model, so quantization restores forgotten knowledge; SURE counters this by updating only saliency-selected modules with a larger learning rate.

Key results

• For unlearning methods with utility constraints, the unlearned model retains on average 21% of forgotten knowledge in full precision, rising to 83% after 4-bit quantization.
• On MUSE BOOKS, GA_KLR's VerMem rises from 13.0 (full precision) to 75.6 after 4-bit RTN quantization.
• SURE substantially improves forgetting after 4-bit quantization (e.g., GA_KLR+SURE+4bit VerMem 16.4 vs 75.6 without SURE on BOOKS) while keeping full-precision utility comparable.

Measured effects reported in the paper

Surprising findings

• 8-bit quantization preserves unlearning performance, while 4-bit quantization causes recovery, and GPTQ/AWQ calibration does not avoid the failure.
• Quantization can appear to improve forgetting for RT on RWKU only because quantization degrades utility rather than removing knowledge.

Limitations

• SURE is highly sensitive to hyperparameter selection, yielding unstable unlearned models.
• Large learning rate on retain data can bias the model and reduce utility on GA_GDR (truthfulness and factuality drop).
• Only RTN, GPTQ, and AWQ at 4- and 8-bit were tested; 2-bit excluded.

Experimental setup

Datasets: MUSE NEWS MUSE BOOKS RWKU WikiText calibration (for GPTQ/AWQ)

Benchmarks: MUSE NEWS MUSE BOOKS RWKU

Metrics: VerMem (ROUGE) KnowMem on forget set PrivLeak (Min-K% AUC) KnowMem on retain set MMLU TruthfulQA MC1 TriviaQA F1 AlpacaEval fluency entropy

Baselines: GA GA_GDR GA_KLR NPO NPO_GDR NPO_KLR RT

Builds on: gradient ascent unlearning negative preference optimization MUSE benchmark TOFU benchmark AWQ

Closest papers by meaning

• Towards Effective Evaluations and Comparisons for LLM Unlearning Methods (ICLR 2025 · similarity 0.93)
• A Closer Look at Machine Unlearning for Large Language Models (ICLR 2025 · similarity 0.93)
• Large Language Model Unlearning (NeurIPS 2024 · similarity 0.92)
• Towards Robust and Parameter-Efficient Knowledge Unlearning for LLMs (ICLR 2025 · similarity 0.92)
• In-Context Unlearning: Language Models as Few-Shot Unlearners (ICML 2024 · similarity 0.92)
• LLM Unlearning via Loss Adjustment with Only Forget Data (ICLR 2025 · similarity 0.92)