Research map / Large language models / Instruction and fine-tuning
Overtrained Language Models Are Harder to Fine-Tune
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that extending pre-training can make LLMs harder to fine-tune ("catastrophic overtraining"), explained by progressive sensitivity of pre-trained parameters, supported by OLMo experiments, controlled small-model experiments, and a deep linear network theory.
Problem
Whether longer pre-training (larger token budgets) always improves downstream performance after post-training; the paper shows extended pre-training can degrade performance after fine-tuning.
Approach
Identifies "catastrophic overtraining" and "progressive sensitivity": for a fixed perturbation magnitude or fixed fine-tuning learning rate, the pre-trained model's perplexity degradation grows monotonically with pre-training tokens, so beyond an inflection point, more pre-training hurts post-fine-tuning performance; theoretically analyzed with two-layer linear networks.
Key results
- Instruction-tuned OLMo-1B pre-trained on 3T tokens performs over 2% worse on multiple standard benchmarks than its 2.3T-token counterpart.
- Under a fixed learning rate, the C4 perplexity increase after fine-tuning grows monotonically with pre-training tokens in controlled 15M-90M models, producing a U-shaped post-fine-tuning perplexity curve.
- OLMo-7B models show no catastrophic overtraining up to 3T pre-training tokens in the reported setups.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Extend pre-training from 2.3T to 3T tokens before Anthropic-HH instruction tuning | OLMo-1B · AlpacaEval | response rate: 3% worse | Hurt |
| Extend pre-training from 2.3T to 3T tokens before Anthropic-HH instruction tuning | OLMo-1B · ARC | performance: 2% worse | Hurt |
| Extend pre-training beyond 2.5T tokens before instruction tuning | OLMo-1B · instruction-tuning ID and OOD tasks | performance | Hurt |
| Extend pre-training beyond 2.5T tokens before multimodal fine-tuning | OLMo-1B · PIQA | performance | Hurt |
| Extend pre-training beyond 2.5T tokens before multimodal fine-tuning | OLMo-1B · VLM score | performance | No clear effect |
| Extend pre-training up to 3T tokens before fine-tuning | OLMo-7B · reported fine-tuning and evaluation setups | performance | No clear effect |
| Increase the pre-training token budget | OLMo-1B base models · downstream tasks before fine-tuning | performance | Helped |
| Increase the pre-training token budget while keeping Gaussian perturbation magnitude fixed | pre-trained models · C4 | perplexity change after perturbation | Hurt |
| Increase the Gaussian perturbation magnitude | pre-trained models · C4 | perplexity after perturbation | Hurt |
| Increase pre-training tokens while holding fine-tuning learning rate fixed | pre-trained models · C4 | perplexity change after fine-tuning | Hurt |
| Use a larger fine-tuning learning rate | pre-trained models · C4 | perplexity after fine-tuning | Hurt |
| Use a larger fine-tuning learning rate, up to a point | pre-trained models · fine-tuning ID task | perplexity | Helped |
Surprising findings
- Extended pre-training always improves base models yet can degrade post-fine-tuned performance, dropping to the level of models pre-trained on 1.5T tokens.
- Regularization and learning rate tuning can delay but not always prevent catastrophic overtraining, and may trade off downstream performance.
Limitations
- Degradation is shown mainly for OLMo-1B; OLMo-7B and LLM360-Amber did not show it within tested budgets.
- Theoretical analysis is limited to two-layer linear networks with gradient flow and Gaussian inputs.
- Focuses on perplexity and a limited set of modifications (fine-tuning and Gaussian noise); RL and pruning left to future work.
Experimental setup
Datasets: Anthropic-HH TULU LLaVA visual instruction tuning data C4
Benchmarks: AlpacaEval ARC-Easy ARC-Challenge HellaSwag PIQA Winogrande
Metrics: AlpacaEval response rate accuracy VLM score C4 perplexity ID perplexity
Baselines: OLMo-1B intermediate checkpoints LLM360-Amber-7B OLMo-2-7B
Builds on: LLaVA Chinchilla scaling laws incremental feature learning in deep linear networks
Closest papers by meaning
- An Empirical Study on Noisy Data and LLM Pretraining Loss Divergence (NeurIPS 2026 · similarity 0.89)
- An Emulator for Fine-tuning Large Language Models using Small Language Models (ICLR 2024 · similarity 0.89)
- Language models scale reliably with over-training and on downstream tasks (ICLR 2025 · similarity 0.88)
- LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws (ICML 2025 · similarity 0.88)
- Selecting Large Language Model to Fine-tune via Rectified Scaling Law (ICML 2024 · similarity 0.87)
- Distilled Pretraining: A modern lens of Data, In-Context Learning and Test-Time Scaling (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.