Research map / Label-efficient and robust learning / Recognition and long-tail

Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods

ICML 2025 · Label-efficient and robust learning › Recognition and long-tail · approach: Data augmentation and generalization bias · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that ensembling pretrained and SFT-ed LLM weights mitigates overadaptation and forgetting, and gives an over-parameterized linear-regression bias-variance theory explaining why.

Problem

Supervised fine-tuning of foundation models causes overadaptation (overfitting to fine-tuning data), which degrades both fine-tuning-domain performance and retention of pretraining knowledge; theory for why weight ensembling helps in over-parameterized models is lacking.

Approach

Interpolate pretrained and fine-tuned weights (model ensembling) and analyze it via a bias-variance decomposition in an over-parameterized linear regression setting, where ridgeless or ridge fine-tuning overfits and a weighted average of pretrained and fine-tuned estimators improves both fine-tuning and pretraining excess risk.

Key results

• On MT-Bench with Dolly fine-tuning, Avg-Norm-Penalty (ensembled) improves over Vanilla-FT for Llama-3-8B (5.96 vs 5.68) and Qwen2-7B (7.10 vs 6.57).
• Ensembling pretrained and fine-tuned weights improves the trade-off between MT-Bench and MMLU/Commonsense-QA, retaining pretraining knowledge while matching or exceeding fine-tuned instruction-following.
• Theorem shows in over-parameterized linear regression that ensemble estimators reduce both fine-tuning excess risk and the sum of pretraining and fine-tuning risk relative to ridgeless and ridge fine-tuned estimators.

Measured effects reported in the paper

Surprising findings

• The ensemble of pretrained and fine-tuned models can outperform the fine-tuned model even on the fine-tuning domain itself (overadaptation).
• Norm-Penalty consistently outperforms DiffNorm-Penalty in almost all settings, suggesting regularizer choice matters.

Limitations

• Theory relies on linear over-parameterized regression with strong assumptions (ordering conditions, subgaussian features) and is only linked to LLM fine-tuning via an NTK heuristic.
• Empirical evaluation uses a single SFT dataset (Dolly) and a small number of models; MT-Bench uses GPT-4 as judge.
• Reported MT-Bench gains are small (often under 0.3 points) and only a few seeds are reported.

Experimental setup

Datasets: Dolly MT-Bench MMLU Commonsense-QA

Benchmarks: MT-Bench MMLU Commonsense-QA

Metrics: MT-Bench score (1-10) accuracy

Baselines: Vanilla-FT Norm-Penalty DiffNorm-Penalty LoRA

Builds on: LP-FT LoRA WiSE-FT

Closest papers by meaning

• LEVI: Generalizable Fine-tuning via Layer-wise Ensemble of Different Views (ICML 2024 · similarity 0.89)
• PACE: Marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularization (NeurIPS 2024 · similarity 0.88)
• Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks (ICLR 2024 · similarity 0.88)
• Upweighting Easy Samples in Fine-Tuning Mitigates Forgetting (ICML 2025 · similarity 0.87)
• The Best Instruction-Tuning Data are Those That Fit (NeurIPS 2025 · similarity 0.87)
• Fine-Tuning is Fine, if Calibrated (NeurIPS 2024 · similarity 0.87)