Research map / Large language models / Alignment and preferences

Large Language Model Unlearning

NeurIPS 2024 · Large language models › Alignment and preferences · approach: Steering and correcting LLM behavior · 34 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes gradient-ascent-based LLM unlearning using only negative examples to remove harmful responses, copyrighted text, and hallucinations, with a random-mismatch loss and KL utility regularization, and compares it to RLHF at about 2% of its compute.

Problem

Aligning LLMs requires costly positive (human-written, helpful) examples in RLHF; practitioners need a cheap way to remove undesirable behaviors (harmful outputs, copyrighted content, hallucinations) from already-trained LLMs without retraining from scratch.

Approach

Apply gradient ascent on undesirable (negative) prompt-output pairs, augmented with a random-mismatch loss that pairs prompts with irrelevant responses and a KL term on normal data (with format matched to the forget data) to preserve utility, so the model stops producing undesirable outputs using only negative samples.

Key results

• GA and GA+Mismatch reduce harmful rate on unlearned harmful prompts for OPT-1.3B from 47% to 1% (GA) and 6% (GA+Mismatch) while preserving normal-prompt utility.
• Unlearning copyrighted Harry Potter text drives leak rate to 0% for OPT-1.3B, OPT-2.7B and Llama 2 (7B) on unlearned extraction attempts.
• Unlearning achieves 4% vs 4% harmful rate on unlearned prompts compared with full RLHF, at about 2% of RLHF's runtime on a single A100 (OPT-1.3B).

Measured effects reported in the paper

Surprising findings

• Gradient ascent loss can rise to ~60 while the model still outputs harmful text, so loss is a weak indicator of unlearning effectiveness; continued unlearning for 3-5x more batches was needed.
• Finetuning on BookCorpus raised the leak rate of OPT-1.3B on HP prompts (78% vs 15% original) because it produced hallucinated HP-related text.

Limitations

• Unlearned outputs are nonsensical strings or whitespace rather than helpful responses; the method only stops undesirable outputs.
• Hallucination reduction is harder: hallucination rate stays around 9-15% rather than near zero.
• Evaluation relies on the original LLM as a proxy for retrained ground truth and on a harmfulness classifier trained on the unlearned data; MIA not evaluated.

Experimental setup

Datasets: PKU-SafeRLHF TruthfulQA BookCorpus Harry Potter and the Sorcerer's Stone HaluEval

Benchmarks: PKU-SafeRLHF TruthfulQA BookCorpus HaluEval

Metrics: harmful rate leak rate hallucination rate diversity fluency (perplexity) utility reward BLEURT similarity to original BLEU BERTScore

Baselines: Finetuning SFT Full RLHF GA GA+Mismatch

Builds on: gradient ascent unlearning label smoothing RLHF

Closest papers by meaning

• A Closer Look at Machine Unlearning for Large Language Models (ICLR 2025 · similarity 0.92)
• Catastrophic Failure of LLM Unlearning via Quantization (ICLR 2025 · similarity 0.92)
• Towards Effective Evaluations and Comparisons for LLM Unlearning Methods (ICLR 2025 · similarity 0.92)
• Keeping an Eye on LLM Unlearning: The Hidden Risk and Remedy (NeurIPS 2025 · similarity 0.92)
• In-Context Unlearning: Language Models as Few-Shot Unlearners (ICML 2024 · similarity 0.92)
• LLM Unlearning via Loss Adjustment with Only Forget Data (ICLR 2025 · similarity 0.92)