Research map / Large language models / Language, safety and interpretability
A Probabilistic Perspective on Unlearning and Alignment for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a probabilistic evaluation framework for LLMs with high-probability bounds on leakage distributions, plus an entropy-optimization unlearning loss and adaptive temperature scaling that reduce sampled leakage in unlearning.
Problem
Deterministic greedy-decoding evaluations of LLMs fail to capture the full output distribution, so they underestimate information leakage in unlearning and toxicity risks under sampling.
Approach
Treat leakage as a random variable over sampled outputs and bound its CDF, probability of leakage, expectation and standard deviation with DKW and Clopper-Pearson guarantees; add an entropy-optimization loss on forget/retain sets and a confidence-based adaptive temperature to make NPO unlearning robust under sampling.
Key results
- On TOFU (90/10 split), NPO with the proposed method reduces probabilistic ED score to 0.20 versus 0.34 for NPO, while greedy ROUGE-L for NPO is 0.22 and the proposed method 0.20.
- On Harry Potter Q&A, greedy decoding indicates successful unlearning but the probabilistic bound shows 38% of questions have expected leakage upper bound above 10%.
- Sampling with 1024 generations shows expected toxicity under sampling consistently exceeds greedy toxicity across Phi-1.5, Vicuna-7b-1.5 and Mistral-7b-instruct-v0.3 on JBB.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add entropy optimization and adaptive temperature scaling to NPO | NPO on Phi-1.5 · TOFU 90/10 split | ED score: 0.34 -> 0.20 | Helped |
| Add entropy optimization and adaptive temperature scaling to NPO | NPO on Phi-1.5 · TOFU 90/10 split | Std. Dev. ROUGE-L: 0.06 -> 0.00 | Helped |
| Evaluate sampled outputs instead of only greedy decoding | Unlearned LLMs · TOFU | ROUGE-L leakage | Mixed |
| Add entropy optimization to NPO | NPO on TOFU forget queries · TOFU | ROUGE-L leakage risk | Helped |
| Increase forget-set entropy regularization strength (ablation) | NPO entropy optimization · TOFU 90/10 and 95/5 splits | Standard deviation of leakage | Helped |
| Lower the sampling temperature (ablation) | Unlearned LLM · TOFU | Standard deviation of ROUGE-L | Helped |
| Add negatively weighted retain-set entropy regularization (ablation) | NPO entropy optimization · TOFU 99/1 split | Output diversity | Helped |
| Vary entropy regularization parameters (ablation) | NPO on TOFU · TOFU Real Authors and World Facts | Model utility | No clear effect |
| Set λf=1 and vary λr (ablation) | Entropy-regularized NPO · TOFU 99/1 split | Utility: 0.496, 0.509, 0.504, 0.490 | No clear effect |
Surprising findings
- Greedy-decoding evaluations of unlearned models can show near-zero leakage while sampling reveals substantial residual information.
- Models with identical greedy ROUGE-L can have very different sampled leakage distributions.
Limitations
- Evaluates output distributions only for fixed inputs, not holistically over all possible inputs.
- Bounds are computed for a single fixed input prompt; computational constraints limit coverage.
- Adaptive temperature uses a hard threshold rather than a more sophisticated scheme.
Experimental setup
Datasets: TOFU Harry Potter Q&A JailbreakBench harmful behaviors
Benchmarks: TOFU Harry Potter Q&A JailbreakBench (JBB)
Metrics: ROUGE-L self-BLEU binary leakage bound general leakage bound expectation bound standard deviation bound ED score model utility
Baselines: Gradient Ascent Gradient Difference RMU NPO greedy decoding evaluation
Builds on: NPO TOFU Clopper-Pearson confidence interval Dvoretzky-Kiefer-Wolfowitz inequality RMU
Closest papers by meaning
- Towards Effective Evaluations and Comparisons for LLM Unlearning Methods (ICLR 2025 · similarity 0.91)
- Constrained Entropic Unlearning: A Primal-Dual Framework for Large Language Models (NeurIPS 2025 · similarity 0.90)
- A Closer Look at Machine Unlearning for Large Language Models (ICLR 2025 · similarity 0.90)
- WAGLE: Strategic Weight Attribution for Effective and Modular Unlearning in Large Language Models (NeurIPS 2024 · similarity 0.90)
- Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation (NeurIPS 2025 · similarity 0.90)
- Toward Adaptive Large Language Models Structured Pruning via Hybrid-grained Weight Importance Assessment (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.