Research map / Large language models / Alignment and preferences
Learning Dynamics of LLM Finetuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives a per-step learning-dynamics decomposition for LLM finetuning (SFT, DPO and variants), explains hallucination and repetition via eNTK-based influence, and identifies a "squeezing effect" from negative gradients that motivates a simple SFT data-extension fix for DPO.
Problem
Understanding how finetuning updates on specific training responses change an LLM's predicted probabilities on other responses, explaining counterintuitive behaviors such as hallucination amplification, repetition, and confidence decay of all responses during off-policy DPO.
Approach
Decomposes one-step change in log-probabilities into an activation/softmax term, an empirical neural tangent kernel between observed and updated inputs, and a residual gradient term; shows that negative gradients on unlikely responses squeeze probability mass into the currently most-likely outputs, and uses this to motivate augmenting SFT data with rejected responses before DPO.
Key results
- Extending SFT with rejected responses before DPO raises win rate against baseline to 0.6928 vs ChatGPT at DPO epoch 4 (from 0.4729 at epoch 0).
- Against Claude3, the extended pipeline reaches 0.6045 win rate at DPO epoch 4.
- Greedy-decoded response log-likelihood rises from -113 to -63 during DPO, faster than SFT increase of the preferred response (-130 to -90).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train on both preferred and rejected responses during SFT, then run DPO | Extend pipeline versus baseline SFT followed by DPO · ChatGPT judge, DPO 2, 4, and 6 epochs | win rate against baseline: 0.6518; 0.6928; 0.6667 | Helped |
| Train on both preferred and rejected responses during SFT, then run DPO | Extend pipeline versus baseline SFT followed by DPO · Claude judge, DPO 2, 4, and 6 epochs | win rate against baseline: 0.5151; 0.6045; 0.5432 | Helped |
| Train on both preferred and rejected responses during SFT, then run DPO | Extend pipeline versus baseline SFT followed by DPO · ChatGPT and Claude judges, DPO 0 epochs | win rate against baseline: 0.4729; 0.4679 | Hurt |
Surprising findings
- Confidence of all tracked responses, including preferred ones, decays during off-policy DPO, which the squeezing effect explains.
- Responses from ChatGPT are all mutually similar in the learned kernel regardless of semantics, suggesting an LLM 'fingerprint'.
Limitations
- Framework relies on teacher forcing and a relatively stable eNTK assumption, so it does not directly cover token-level RLHF/PPO.
- Squeezing effect analysed for logistic regression and only qualitatively extended to autoregressive DPO.
- Win-rate evaluation uses only ChatGPT and Claude3 as judges on a limited scale.
Experimental setup
Datasets: Anthropic-HH UltraFeedback
Metrics: win rate log-likelihood of response types
Baselines: baseline SFT followed by DPO
Builds on: direct preference optimization neural tangent kernel TracIn
Closest papers by meaning
- What Do Learning Dynamics Reveal About Generalization in LLM Mathematical Reasoning? (ICML 2025 · similarity 0.89)
- LensLLM: Unveiling Fine-Tuning Dynamics for LLM Selection (ICML 2025 · similarity 0.89)
- Dynamics-Predictive Sampling for Active RL Finetuning of Large Reasoning Models (ICLR 2026 · similarity 0.89)
- Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data (ICML 2024 · similarity 0.88)
- Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections (NeurIPS 2025 · similarity 0.88)
- Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.