Research map / Large language models / Instruction and fine-tuning
Model Collapse Is Not a Bug but a Feature in Machine Unlearning for LLMs
Yan Scholten, Sophie Xhonneux, Leo Schwinn, Stephan Günnemann
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Partial Model Collapse unlearning instead of target-based baselines | Phi-1.5, Llama-3.2-3B-Instruct, Gemma-3-12b-it · TOFU | utility vs unlearn quality Pareto front | Improved |
| replace NPO target-based unlearning with PMC | Llama / Phi · Wikitext token probabilities | probability difference of forget tokens: -0.12 mean for NPO vs zero-mean for PMC | Improved |
| replace baselines with PMC under sampling and prefilling attacks | unlearned models · TOFU forget set | worst-case ROUGE-L leakage | Improved |
| train more epochs (up to 20) (ablation) | PMC · TOFU (Phi-1.5) | unlearn quality | Improved |
| increase number of samples (ablation) | PMC · TOFU (Phi-1.5) | unlearn quality | Improved |
| increase lambda trade-off (ablation) | PMC · TOFU (Phi-1.5) | utility and unlearn quality | Mixed |
| increase sampling temperature (ablation) | PMC · TOFU (Phi-1.5) | unlearn quality and utility | Mixed |
| use fewer than 10 samples (ablation) | PMC · TOFU (Phi-1.5) | collapse to max reward | Hurt |
| use self-BLEU reward instead of ROUGE-L reward (ablation) | PMC · TOFU (Llama-3.2-3B-Instruct) | utility | Improved |
| add PMC fine-tuning (vs base) on general benchmarks | Phi-1.5 · Arc-Easy | accuracy: 0.7622 -> 0.6891 | Hurt |
| add PMC fine-tuning (vs base) on general benchmarks | Llama-3.2-3B-Instruct · MMLU | accuracy: 0.6041 -> 0.5924 | No clear effect |
| use gradient checkpointing (ablation) | PMC · TOFU | unlearning trajectory | Hurt |
Closest papers by meaning
- The Utility and Complexity of In- and Out-of-Distribution Machine Unlearning (ICLR 2025 · similarity 0.91)
- Unlearning or Obfuscating? Jogging the Memory of Unlearned LLMs via Benign Relearning (ICLR 2025 · similarity 0.91)
- Unlearned but Not Forgotten: Data Extraction after Exact Unlearning in LLM (NeurIPS 2025 · similarity 0.90)
- Machine Unlearning Doesn't Do What You Think: Lessons for Generative AI Policy and Research (NeurIPS 2025 · similarity 0.90)
- LLM Unlearning via Loss Adjustment with Only Forget Data (ICLR 2025 · similarity 0.90)
- Keeping an Eye on LLM Unlearning: The Hidden Risk and Remedy (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
