Research map / Large language models / Language, safety and interpretability

LLM Unlearning via Loss Adjustment with Only Forget Data

ICLR 2025 · Large language models › Language, safety and interpretability · approach: LLM unlearning and benchmarks · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FLAT, an LLM unlearning method that uses only forget data (plus template answers) and an f-divergence variational loss adjustment, avoiding retain data and reference models.

Problem

LLM unlearning typically needs retain data or a reference model to preserve utility, which is often unavailable or costly and blurs the boundary between forget and retain responses, causing a utility-forgetting trade-off.

Approach

Maximize the f-divergence between template (e.g., IDK) answers and original forget answers using only forget data; the variational form yields a loss adjustment that weights learning from template responses against forgetting, with no retain data or reference model.

Key results

• On Harry Potter with OPT-2.7B, FLAT (KL) achieves FQ Gap 1.3238 (second-best behind NPO's 1.2611) with PPL 15.5311 and average accuracy 0.4694, close to the retained model's 0.4686.
• On TOFU, FLAT (KL) attains the best model utility on Llama2-7B (0.6393 vs retained 0.6267) with forget quality 0.0286, in the top two.
• On MUSE-News, FLAT (TV) gets the lowest VerbMem (1.7) and KnowMem (13.6) on the forget set among FLAT variants, but none of the methods meet the PrivLeak criterion.

Measured effects reported in the paper

Surprising findings

• Retain data-based fine-tuning and reference-model methods trade off utility against forgetting, while FLAT achieves competitive results with forget data alone.
• On MUSE, all evaluated methods, including FLAT, show substantial privacy leakage as measured by PrivLeak.

Limitations

• Poor performance on privacy leakage (PrivLeak) evaluation on MUSE, shared by all tested methods.
• Performance across the four f-divergence variants is similar, so the benefit of the specific divergence is limited.
• Relies on template responses (e.g., IDK or normal answers) whose choice affects forgetting versus utility.

Experimental setup

Datasets: Harry Potter series C4 TOFU MUSE-News (BBC News) Wikitext IDK responses

Benchmarks: Harry Potter copyrighted content unlearning TOFU MUSE-News Wikitext perplexity nine zero-shot tasks

Metrics: FQ Gap (BLEU and ROUGE-L) PPL zero-shot average accuracy forget quality (KS p-value) model utility ROUGE-L VerbMem KnowMem PrivLeak

Baselines: GA KL GradDiff PO Mismatch LLMU DPO NPO NPO-RT Task Vectors WHP

Builds on: gradient ascent direct preference optimization LLMU NPO Who's Harry Potter

Closest papers by meaning

• Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Difference (NeurIPS 2024 · similarity 0.94)
• A Closer Look at Machine Unlearning for Large Language Models (ICLR 2025 · similarity 0.93)
• Towards Effective Evaluations and Comparisons for LLM Unlearning Methods (ICLR 2025 · similarity 0.92)
• Keeping an Eye on LLM Unlearning: The Hidden Risk and Remedy (NeurIPS 2025 · similarity 0.92)
• Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning (NeurIPS 2025 · similarity 0.92)
• Towards Robust and Parameter-Efficient Knowledge Unlearning for LLMs (ICLR 2025 · similarity 0.92)