Research map / Large language models / Instruction and fine-tuning
Mitigating Forgetting in LLM Fine-Tuning via Low-Perplexity Token Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that fine-tuning on LLM-generated (low-perplexity) data reduces forgetting on non-target tasks, and proposes Selective Token Masking (STM), which masks high-perplexity ground-truth tokens using the initial instruction-tuned model, matching that benefit with LoRA fine-tuning.
Problem
Fine-tuning LLMs on domain tasks degrades performance on non-target tasks (catastrophic forgetting); it is unclear why LLM-generated training data causes less degradation than ground-truth data.
Approach
Attributes reduced forgetting from LLM-generated data to fewer high-perplexity tokens; simply masking the top ~20-25% high-perplexity ground-truth tokens (perplexity computed by the same instruction-tuned model in one forward pass) recovers similar non-target preservation without generating data.
Key results
- STM with threshold 2.5 (about 22-24% of tokens masked) on Llama 3 8B Instruct trained on MBPP yields BWT -0.16% and TI +3.20%, versus baseline fine-tuning BWT -34.71% and TI -2.23%.
- On Gemma 2 IT 2B trained on MBPP, STM reaches BWT +0.42% and TI 0.00% vs baseline BWT -38.19% and TI -21.76%, at about 5 GPU minutes versus 12 GPU hours for Self-Output.
- Masking random or low-perplexity tokens instead of high-perplexity ones fails to preserve non-target performance (e.g., BWT -40.41% random on Llama 3 8B for MBPP), supporting the high-perplexity explanation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Mask high-perplexity tokens in ground-truth training data with STM τ=2.5 instead of baseline fine-tuning | Gemma 2 IT 2B · MBPP target; non-target tasks summarized by BWT | BWT (%): -38.19 -> 0.42 | Helped |
| Mask high-perplexity tokens in ground-truth training data with STM τ=2.5 instead of baseline fine-tuning | Gemma 2 IT 2B · MATH target; non-target tasks summarized by BWT | BWT (%): -36.68 -> -2.93 | Helped |
| Mask high-perplexity tokens in ground-truth training data with STM τ=2.5 instead of baseline fine-tuning | Llama 3 8B Instruct · MBPP target; non-target tasks summarized by BWT | BWT (%): -34.71 -> -0.16 | Helped |
| Mask high-perplexity tokens in ground-truth training data with STM τ=2.5 instead of baseline fine-tuning | Llama 3 8B Instruct · MATH target; non-target tasks summarized by BWT | BWT (%): -14.12 -> -0.30 | Helped |
| Filter the highest-perplexity tokens rather than randomly selected tokens (ablation) | STM on Gemma 2 IT 2B · MBPP target | BWT (%): -8.6 -> 0.4 | Helped |
| Filter the highest-perplexity tokens rather than the lowest-perplexity tokens (ablation) | STM on Gemma 2 IT 2B · MBPP target | TI (%): -18.7 -> 0.0 | Helped |
| Use STM τ=2.5, filtering 23.8% of tokens, rather than τ=1000, filtering 6.26% (ablation) | STM on Gemma 2 IT 2B · MBPP target | BWT (%): -2.9 -> 0.4 | Helped |
| Use perplexities from Gemma 2 IT 9B to select tokens rather than the target model's own 2B perplexities (ablation) | STM on Gemma 2 IT 2B · MBPP target | BWT (%): 0.4 -> -3.8 | Hurt |
| Add STM τ=2.5 to LoRA fine-tuning (ablation) | Gemma 2 IT 2B · MBPP target | BWT (%): -38.19 -> 0.42 | Helped |
| Add STM τ=2.5 to full-weight fine-tuning (ablation) | Gemma 2 IT 2B · MBPP target | BWT (%): -31.87 -> -0.13 | Helped |
| Add STM τ=2.5 to DoRA fine-tuning (ablation) | Llama 3 8B Instruct · MBPP target | TI (%): -8.03 -> 3.58 | Helped |
| Add STM τ=2.5 to LoRA fine-tuning (ablation) | Llama 3 8B Instruct · MBPP target | BWT (%): -34.71 -> -0.3 | Helped |
Surprising findings
- Masking ~24% of high-perplexity ground-truth tokens matches or beats Self-Output data on both target and non-target tasks while requiring only a single forward pass.
- Regularization (dropout, weight decay, KL) with similar weight-update norms does not match STM's non-target preservation, suggesting degradation stems from high-perplexity training rather than overfitting alone.
Limitations
- STM was not applied to models larger than 10B due to resource limits (only 9B tested for cross-scale filtering).
- Non-target evaluation covers only a limited set of tasks (no tool use, medicine, or finance).
- Threshold selection is heuristic (about 20-25% masked), with no principled adaptive masking mechanism.
Experimental setup
Datasets: MBPP MATH GSM8K ARC-Challenge BIRD IFEval AdvBench
Benchmarks: MBPP MATH GSM8K ARC-Challenge BIRD IFEval AdvBench
Metrics: backward transfer (BWT) task improvement (TI) pass@1 accuracy exact match prompt-level strict accuracy refuse rate harmful rate self-BLEU L2 norm of delta W
Baselines: Baseline Fine-tuning Self-Output Rephrase Dropout Weight Decay KL regularization
Builds on: LoRA DoRA Selective Language Modeling self-distillation
Closest papers by meaning
- Learning What to Forget: Improving LLM Unlearning via Learned Token-Level Importance (NeurIPS 2026 · similarity 0.90)
- Upweighting Easy Samples in Fine-Tuning Mitigates Forgetting (ICML 2025 · similarity 0.89)
- Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine-Tuning (ICML 2025 · similarity 0.89)
- Learning and Forgetting Unsafe Examples in Large Language Models (ICML 2024 · similarity 0.89)
- LLM Unlearning via Loss Adjustment with Only Forget Data (ICLR 2025 · similarity 0.88)
- LoRA vs Full Fine-tuning: An Illusion of Equivalence (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.