Research map / Large language models / Instruction and fine-tuning
Divergence Decoding: Inference-Time Unlearning via Auxiliary Models
Humzah Merchant, Bradford Levy
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Divergence Decoding instead of gradient-based unlearning baselines | Divergence Decoding · TOFU | aggregate score | Improved |
| use Divergence Decoding on MUSE Knowledge Memorization | Divergence Decoding · MUSE Knowledge Memorization | distance to Retrain | Improved |
| use Divergence Decoding on MUSE Verbatim Memorization | Divergence Decoding · MUSE Verbatim Memorization | distance to Retrain | Hurt |
| use rank-based DD instead of linear DD (ablation) | Divergence Decoding · MUSE and TOFU | distance / aggregate score | Mixed |
| scale auxiliary models from 1.3B to 2.7B (ablation) | Divergence Decoding on MUSE · MUSE | distance to Retrain | Improved |
| scale auxiliary models from 1B to 3B (ablation) | Divergence Decoding on TOFU · TOFU | aggregate score | Improved |
| use trigram LM auxiliaries instead of small neural LMs (ablation) | Divergence Decoding · TOFU | aggregate score | Hurt |
| use trigram LM auxiliaries instead of small neural LMs (ablation) | Divergence Decoding · MUSE | distance to Retrain | Mixed |
| use auxiliary models from a different family or tokenizer (OLMo-2, Gemma-3, Qwen3) (ablation) | Divergence Decoding · TOFU | aggregate score | No clear effect |
| distill Linear DD teacher into the target model | Divergence Decoding · TOFU and MUSE | unlearning score | Improved |
| apply Distill DD under Leak@K adversarial prompting | Distill DD · TOFU Leak@K | leakage | Improved |
| increase alpha in linear DD for image generation (1 to 5) (ablation) | Linear DD · ImageNet dog forget classes | class presence VQAScore: 20% -> 1% | Mixed |
Closest papers by meaning
- In-Context Unlearning: Language Models as Few-Shot Unlearners (ICML 2024 · similarity 0.90)
- Large Language Model Unlearning via Embedding-Corrupted Prompts (NeurIPS 2024 · similarity 0.90)
- LLM Unlearning via Loss Adjustment with Only Forget Data (ICLR 2025 · similarity 0.90)
- Towards Robust and Parameter-Efficient Knowledge Unlearning for LLMs (ICLR 2025 · similarity 0.90)
- A Closer Look at Machine Unlearning for Large Language Models (ICLR 2025 · similarity 0.89)
- Constrained Entropic Unlearning: A Primal-Dual Framework for Large Language Models (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
