Research map / Large language models / Alignment and preferences
Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Identifies reference-model bias in negative preference optimization (NPO) for LLM unlearning and proposes SimNPO, a reference-free, length-normalized variant that improves forget quality and utility on TOFU, MUSE and WMDP.
Problem
LLM unlearning via negative preference optimization relies on a reference model (the pre-unlearning model), which biases unlearning power allocation across forget samples of differing memorization and length and gives ineffective early-stage gradient weight smoothing.
Approach
Replace NPO's DPO-style reference-model reward with SimPO's reference-free, length-normalized reward (beta/|y|) log pi, yielding a SimNPO loss whose gradient weights depend on the forget sample's length and not on the reference model.
Key results
- On TOFU Forget05 with LLaMA-2-7B-chat, SimNPO reaches FQ 0.99 and MU 0.58 versus NPO's FQ 0.79 and MU 0.57.
- On MUSE Books, SimNPO achieves 0.00 VerbMem and KnowMem on the forget set with retain KnowMem 48.27, compared with NPO's retain KnowMem 23.71.
- SimNPO shows improved robustness over NPO under shortest-20% and random relearning attacks on TOFU Forget05.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace NPO's reference-based objective with reference-free, length-normalized SimNPO | LLM unlearning on TOFU Forget05 · TOFU Forget05 | FQ: 0.79 -> 0.99 | Helped |
| Replace NPO's reference-based objective with reference-free, length-normalized SimNPO | LLM unlearning on TOFU Forget05 · TOFU Forget05 | MU: 0.57 -> 0.58 | Helped |
| Replace NPO with SimNPO | MUSE News unlearning · MUSE News | KnowMem on retain set: 37.58 -> 39.65 | Helped |
| Replace NPO with SimNPO | MUSE News unlearning · MUSE News | KnowMem on forget set: 56.93 -> 44.84 | Helped |
| Replace NPO with SimNPO | MUSE News unlearning · MUSE News | PrivLeak: 108.91 -> 72.93 | Helped |
| Replace NPO with SimNPO | MUSE Books unlearning · MUSE Books | KnowMem on retain set: 23.71 -> 48.27 | Helped |
| Replace NPO with SimNPO | MUSE Books unlearning · MUSE Books | PrivLeak: -31.17 -> -19.82 | Helped |
| Perturb the reference model used by NPO (ablation) | NPO on TOFU Forget05 · TOFU Forget05 | FQ: 0.79 -> 0.27 | Hurt |
| Perturb the reference model used by NPO (ablation) | NPO on TOFU Forget05 · TOFU Forget05 | MU: 0.57 vs. $0.52 | No clear effect |
| Use reference-free SimNPO instead of NPO | Unlearning differently memorized TOFU subsets · TOFU Forget05, strongly memorized subset D_f,1 | FQ: 0.00 -> 0.70 | Helped |
| Set the SimNPO reward margin gamma above 0 (ablation) | SimNPO on TOFU Forget05 · TOFU Forget05 | MU | Hurt |
| Set gamma to 0 rather than a positive reward margin (ablation) | SimNPO on TOFU Forget05 · TOFU Forget05 | balance between FQ and MU | Helped |
Surprising findings
- Setting the reference model to a random perturbation drops NPO's forget quality on TOFU Forget05 from 0.79 to 0.27 while utility stays nearly unchanged.
- NPO leaves short-response forget data poorly unlearned (FQ 0.58) while long-response data reaches FQ 0.81.
Limitations
- Both SimNPO and NPO still rely on promoting divergence, causing some utility loss, especially on WMDP knowledge removal.
- No theoretical guarantees for SimNPO are established.
- Results on TOFU use 5 random trials but the hyperparameter grid searches are extensive and the method introduces additional hyperparameters (gamma, beta).
Experimental setup
Datasets: TOFU MUSE WMDP
Benchmarks: TOFU MUSE WMDP
Metrics: forget quality model utility VerbMem KnowMem PrivLeak accuracy on WMDP-Bio accuracy on MMLU relearning forget quality
Baselines: Retrain GA GradDiff IDK RKLD NPO Task Vector RMU
Builds on: negative preference optimization direct preference optimization SimPO gradient difference
Closest papers by meaning
- LLM Unlearning via Loss Adjustment with Only Forget Data (ICLR 2025 · similarity 0.92)
- Rethinking LLM Unlearning Objectives: A Gradient Perspective and Go Beyond (ICLR 2025 · similarity 0.92)
- WAGLE: Strategic Weight Attribution for Effective and Modular Unlearning in Large Language Models (NeurIPS 2024 · similarity 0.92)
- Towards Effective Evaluations and Comparisons for LLM Unlearning Methods (ICLR 2025 · similarity 0.92)
- A Closer Look at Machine Unlearning for Large Language Models (ICLR 2025 · similarity 0.92)
- Towards Robust and Parameter-Efficient Knowledge Unlearning for LLMs (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.