Research map / Large language models / Language, safety and interpretability

RULE: Reinforcement UnLEarning Achieves Forget-retain Pareto Optimality

NeurIPS 2025 · Large language models › Language, safety and interpretability · approach: LLM unlearning and benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RULE, a two-stage refusal-based unlearning method (rejection steering SFT then on-policy RL with a regex-based verifiable reward over forget and synthesized boundary queries) that forgets target knowledge with small data while keeping response naturalness and utility.

Problem

LLM unlearning methods often produce unnatural or collapsed responses on forget queries, depend on large explicit forget and retain sets, and trade off forgetting against general utility.

Approach

Formulate unlearning as refusal boundary optimization: warm-start with supervised refusal on a small forget set, then run on-policy RL (PPO, GRPO, Reinforce++) over forget queries plus GPT-4o-mini-generated entity-swapped boundary queries, using an intrinsic reward that rewards refusal on forget prompts and informative answers on boundary prompts.

Key results

• On RWKU with llama3-8b-instruct, ReBO_RPP reaches overall Forget Quality 22.6, beating the best baseline SimNPO (40.1) by 17.5 points.
• ReBO_GRPO reaches Forget Naturalness 89.1 versus 72.8 for the best baseline NPO+KLR, a 16.3-point gain.
• ReBO_GRPO reaches Forget Naturalness 89.1 and retains Retain Quality 73.7, while GA and SimNPO collapse to zero AUC when retain is at least 0.6 in the Pareto analysis.

Measured effects reported in the paper

Surprising findings

• Removing the boundary set (training retain on another target's forget set) drives forget quality to 19.9 but collapses naturalness to 25.4 and retain quality to 23.6.
• Rejection steering alone improves truthfulness (94.8) but barely forgets (overall forget quality 57.1).

Limitations

• Relies on synthetic boundary data generated by GPT-4o-mini, which may limit scalability
• Reward uses regex refusal matching, which may miss or misclassify refusal phrasings (about 95% agreement with humans on sampled cases)
• Evaluated only on RWKU and MUSE with llama3 8B models; multi-turn and multilingual settings not tested

Experimental setup

Datasets: RWKU MUSE-books MUSE-news TOFU refusal templates

Benchmarks: RWKU MUSE

Metrics: ROUGE-L forget quality retain quality naturalness (readability, helpfulness, truthfulness) general utility (reasoning, truthfulness, factuality, fluenc Pareto AUC

Baselines: Gradient Ascent GDR KLR NPO SimNPO Rejection Steering only

Builds on: GRPO PPO Reinforce++ NPO SimNPO

Closest papers by meaning

• LLM Unlearning via Loss Adjustment with Only Forget Data (ICLR 2025 · similarity 0.91)
• Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning (NeurIPS 2025 · similarity 0.91)
• RA$^2$: Retain-Anchored Attraction Preserves Forget-Adjacent Utility in LLM Unlearning (NeurIPS 2026 · similarity 0.91)
• Constrained Entropic Unlearning: A Primal-Dual Framework for Large Language Models (NeurIPS 2025 · similarity 0.91)
• What to Forget in Unlearning? Forget Set Curation for Language Models (NeurIPS 2026 · similarity 0.91)
• Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Difference (NeurIPS 2024 · similarity 0.91)