Research map / Large language models / Agents, code and math
Maximizing Intermediate Checkpoint Value in LLM Pretraining with Bayesian Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes merging adjacent intermediate LLM pretraining checkpoints with a weight chosen by Bayesian optimization (GP + GP-Hedge) on a held-out set, reporting gains over soup baselines on Baichuan2, DeepSeek and Pythia checkpoints.
Problem
Intermediate checkpoints saved during LLM pretraining are underused; naive checkpoint averaging is high-dimensional and the best merging weight is unclear, so the goal is to extract more performance from them at minimal cost.
Approach
Pairwise merging of the two most recent checkpoints reduces the problem to a one-dimensional search over the merging weight, which is solved with Gaussian-process Bayesian optimization using a GP-Hedge combination of EI, PI and UCB acquisition functions, evaluated on a held-out dataset.
Key results
- On Baichuan2-1980B+2200B, the Bayesian-optimized merge reaches 56.17 on C-Eval and 56.88 on CMMLU, averaging 48.13 vs 47.65 for the best baseline (Greedy Soup).
- On Pythia 70M PIQA, the method scores 60.18%, 1.22 points above the best baseline (RegMean).
- Out-of-domain gap across CMMLU, MMLU and GSM8K is 0.87 for the proposed method vs 2.01 (Greedy Soup) and 2.38 (Fisher).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Bayesian-optimized pairwise checkpoint merging instead of the strongest reported merging baseline | Baichuan2-1980B + Baichuan2-2200B · C-Eval (5-shot) | accuracy: 55.73 -> 56.17 | Helped |
| Use Bayesian-optimized pairwise checkpoint merging instead of the strongest reported merging baseline | Baichuan2-1980B + Baichuan2-2200B · CMMLU (5-shot) | accuracy: 56.13 -> 56.88 | Helped |
| Use Bayesian-optimized pairwise checkpoint merging instead of the strongest reported merging baseline | Baichuan2-1980B + Baichuan2-2200B · MMLU (5-shot) | accuracy: 55.39 -> 55.44 | Helped |
| Use Bayesian-optimized pairwise checkpoint merging instead of the strongest reported merging baseline | Baichuan2-1980B + Baichuan2-2200B · GSM8K (4-shot) | accuracy: 23.96 -> 24.02 | Helped |
| Use Bayesian-optimized pairwise checkpoint merging instead of the strongest reported merging baseline | Baichuan2-2200B + Baichuan2-2420B · C-Eval (5-shot) | accuracy: 55.64 -> 56.23 | Helped |
| Use Bayesian-optimized pairwise checkpoint merging instead of the strongest reported merging baseline | Baichuan2-2200B + Baichuan2-2420B · CMMLU (5-shot) | accuracy: 56.78 -> 56.97 | Helped |
| Use Bayesian-optimized pairwise checkpoint merging instead of the strongest reported merging baseline | Baichuan2-2200B + Baichuan2-2420B · MMLU (5-shot) | accuracy: 54.82 -> 54.56 | Hurt |
| Use Bayesian-optimized pairwise checkpoint merging instead of the strongest reported merging baseline | Baichuan2-2200B + Baichuan2-2420B · GSM8K (4-shot) | accuracy: 23.88 -> 24.32 | Helped |
| Use Bayesian-optimized merging rather than the best reported baseline | Pythia models · PIQA (5-shot), across model sizes 70M to 6.9B | accuracy: 58.96 -> 60.18; 68.32 -> 68.85; 71.04 -> 71.84; 74.97 -> 75.71; 76.44 -> 76.56 | Helped |
| Use Bayesian optimization instead of random, greedy, or grid search for merging weights | Baichuan2-1980B + Baichuan2-2200B · C-Eval, CMMLU, MMLU, and GSM8K | accuracy: 56.73(+1.09); 57.05(+0.31); 54.77(+0.32); 22.17(+0.34) | Helped |
| Use 1/4 of the held-out validation data rather than the full validation data (ablation) | Bayesian-optimized merging of Baichuan2-2200B + Baichuan2-2420B · C-Eval | accuracy: 56.23 -> 56.61 | No clear effect |
| Use 1/2 of the held-out validation data rather than the full validation data (ablation) | Bayesian-optimized merging of Baichuan2-2200B + Baichuan2-2420B · C-Eval | accuracy: 56.23 -> 56.08 | No clear effect |
Surprising findings
- Merging a checkpoint with a nearby earlier checkpoint can beat the stronger final checkpoint, while merging distant checkpoints severely degrades performance (e.g., 25.26% vs 54.98%).
Limitations
- Requires a held-out validation dataset to select merging weights, and the gains are small (often under 1 point).
- Restricted to pairwise merging of adjacent checkpoints; multi-checkpoint merging performed worse in the experiments.
- Gains are reported as single-run comparisons with no variance or significance analysis.
Experimental setup
Datasets: C-Eval validation set (held-out for merging weights)
Benchmarks: C-Eval CMMLU MMLU GSM8K PIQA WinoGrande SciQ ARC-Easy
Metrics: accuracy average accuracy absolute difference between in-domain and out-of-domain accu
Baselines: Uniform Soup Greedy Soup Fisher Weighted Averaging RegMean Random Search Greedy Search Grid Search
Builds on: Model soups (Wortsman et al., 2022) Fisher merging (Matena & Raffel, 2022) RegMean GP-Hedge
Closest papers by meaning
- Fast and Accurate Probing of In-Training LLMs' Downstream Performances (NeurIPS 2026 · similarity 0.88)
- Model Merging in Pre-training of Large Language Models (NeurIPS 2025 · similarity 0.88)
- Benchmarking Optimizers for Large Language Model Pretraining (NeurIPS 2026 · similarity 0.88)
- MergeBench: A Benchmark for Merging Domain-Specialized LLMs (NeurIPS 2025 · similarity 0.88)
- SkyLadder: Better and Faster Pretraining via Context Window Scheduling (NeurIPS 2025 · similarity 0.88)
- Reinforcement Mid-Training (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.