Research map / Label-efficient and robust learning / Recognition and long-tail

Contrastive Representation Shaping for LLM Unlearning

NeurIPS 2026 · Label-efficient and robust learning › Recognition and long-tail · approach: Contrastive representation theory · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CLReg, a contrastive regularizer that pushes forget-example embeddings away from retain embeddings and combines with five LLM unlearning methods, improving UnlearningScore on TOFU, MUSE, and WMDP.

Problem

Approximate LLM unlearning methods suppress forgotten outputs by aligning prediction distributions, but forgotten concepts persist in representations entangled with retained knowledge, making forgetting shallow and easy to reverse.

Approach

Adds a contrastive loss on pooled hidden states of a late layer: forget anchors are pulled toward dropout/paraphrase augmentations of themselves and pushed away from retain embeddings as negatives, as a plug-in regularizer on top of any base unlearning objective, without labels.

Key results

• SimNPO+CL reaches UnlearningScore 0.813 on TOFU Llama-3-8B versus 0.774 for SimNPO and 0.704 for RMU.
• CLReg improves UnlearningScore for all five base methods across 25 method-benchmark pairs, with GradDiff+CL lifting MUSE-Books from 0.411 to 0.738.
• CLReg cuts forget-retain entanglement by up to 88% (SimNPO on MUSE-News, 92.2 to 11.1).

Surprising findings

• Applying only the CL objective with no forgetting loss (stage 1 warm start) makes a subsequent unmodified base method unlearn more effectively.
• CLReg makes relearning harder for SimNPO and GradDiff, while NPO barely relearns at baseline.

Limitations

• Proposition-level analysis is one-step and only motivational; no multi-step or retain-side guarantees.
• Layer choice is fixed to late layers; concepts localized in earlier layers may need different layers.
• Fixed lambda=1 causes some over-unlearning and privacy-leak regressions (e.g., GradDiff+CL on MUSE-News).

Experimental setup

Datasets: TOFU MUSE-Books MUSE-News WMDP-cyber

Benchmarks: TOFU MUSE WMDP-cyber

Metrics: UnlearningScore ForgetScore ModelUtility ForgetQuality PrivLeak MK-MMD 2-Wasserstein variance-based entanglement relearning recovery AUC

Baselines: GradDiff UNDIAL NPO SimNPO PDU RMU

Builds on: NPO SimNPO SimCSE DPO RMU

Closest papers by meaning

• RA$^2$: Retain-Anchored Attraction Preserves Forget-Adjacent Utility in LLM Unlearning (NeurIPS 2026 · similarity 0.89)
• Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Difference (NeurIPS 2024 · similarity 0.89)
• CoUn: Empowering Machine Unlearning via Contrastive Learning (NeurIPS 2025 · similarity 0.88)
• From Dormant to Deleted: Tamper-Resistant Unlearning Through Weight-Space Regularization (NeurIPS 2025 · similarity 0.88)
• Learning What to Forget: Improving LLM Unlearning via Learned Token-Level Importance (NeurIPS 2026 · similarity 0.88)
• Unlearning or Obfuscating? Jogging the Memory of Unlearned LLMs via Benign Relearning (ICLR 2025 · similarity 0.87)