Research map / Large language models / Instruction and fine-tuning

Preserving Diversity in Supervised Fine-Tuning of Large Language Models

ICLR 2025 · Large language models › Instruction and fine-tuning · approach: Fine-tuning generalization and delta compression · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GEM, a game-theoretic SFT objective with an auxiliary meta-controller distribution that equals a softmax of the model's logits scaled by 1/beta, which preserves output diversity and reduces forgetting relative to cross-entropy fine-tuning.

Problem

Cross-entropy SFT of LLMs maximizes likelihood of observed responses, collapsing output diversity and causing knowledge forgetting and alignment tax, which hurts later sampling-based stages such as test-time scaling.

Approach

Frame SFT as a two-player game with an auxiliary distribution q that governs logit flow from high-probability source tokens to the target; q is a softmax of shifted logits, so the equilibrium is reverse-KL matching with entropy regularization, computed at the same cost as CE.

Key results

• GEM reaches a 3.1-point (6.6% relative) higher win rate than CE in best-of-32 chat evaluation on AlpacaEval, with about 2x fewer samples needed for comparable performance.
• On HumanEval pass@100, GEM reaches 82.5 versus 75.5 for CE, a 7-point gain.
• On the OpenLLM leaderboard tasks, GEM drops 0.3 points on average versus 1.5 for CE, reported as an 80% reduction in alignment tax.

Measured effects reported in the paper

Surprising findings

• Adding an entropy regularizer directly to CE produces nonsensical tail tokens in generated text, whereas GEM does not.
• Weight decay does not improve best-of-N chat performance and only slightly improves CE on code pass@100.

Limitations

• beta = 0 case lacks a unique tractable equilibrium and is left for future work
• Reported results are mainly on Llama-3.1-8B with UltraFeedback, so generality across models is shown only partially in the excerpt
• Chat evaluation relies on a reward model as selector rather than human judgments

Experimental setup

Datasets: UltraFeedback AlpacaEval HumanEval OpenLLM leaderboard tasks

Benchmarks: AlpacaEval HumanEval OpenLLM leaderboard (ARC, GSM8K, HellaSwag, MMLU, TruthfulQA, WinoGrande) RewardBench

Metrics: win rate pass rate / pass@100 100 - Self-BLEU diversity entropy average accuracy drop (alignment tax)

Baselines: cross-entropy (CE) CE + weight decay NEFT SPIN

Builds on: cross-entropy SFT SPIN GAN weight decay NEFT

Closest papers by meaning

• Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models (ICML 2024 · similarity 0.88)
• Constrained Entropic Unlearning: A Primal-Dual Framework for Large Language Models (NeurIPS 2025 · similarity 0.88)
• Large Language Models to Diffusion Finetuning (ICML 2025 · similarity 0.88)
• Linearization Explains Fine-Tuning in Large Language Models (NeurIPS 2025 · similarity 0.88)
• LensLLM: Unveiling Fine-Tuning Dynamics for LLM Selection (ICML 2025 · similarity 0.88)
• Diversity-Aware Policy Optimization for Large Language Model Reasoning (NeurIPS 2025 · similarity 0.87)