Research map / Large language models / Language, safety and interpretability
Model Merging in Pre-training of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies checkpoint merging (PMA) during LLM pre-training on ByteDance Seed dense and MoE models up to 100B+ parameters, showing merged stable-phase checkpoints approximate annealed performance and can stabilize continued training via PMA-init.
Problem
Model merging has been studied mainly for post-training; its use during large-scale LLM pre-training is underexplored and public detail is scarce, while pre-training costs and loss-spike instability are major challenges.
Approach
Average sequential checkpoints from a single pre-training trajectory (Pre-trained Model Average, PMA) under a constant learning rate in the WSD schedule, so the merged model approximates an annealed model without decaying the learning rate; PMA-init reuses merged weights to initialize CT/SFT and to recover from loss spikes.
Key results
- On Seed-MoE-1.3B/13B, merged checkpoints raised HumanEval from 31.1 to 36.6, and Seed-MoE-10B/100B from 54.3 to 61.6.
- Seed-Dense-70B improved from 50.6 to 57.9 on HumanEval and from 85.9 to 91.3 on GSM8K after merging.
- With PMA-init for SFT, PMA at 2e-5 to 2e-6 improved in-house OOD from 32.6 to 34.7 and Instruction Following from 36.3 to 38.8.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Merge checkpoints from stable training | Seed-MoE models · HumanEval | score: 31.1 -> 36.6; 54.3 -> 61.6 | Helped |
| Merge stable-training checkpoints | Seed-Dense-70B · GSM8K | score: 85.9 -> 91.3 | Helped |
| Use WMA instead of SMA or EMA (ablation) | Seed-MoE-1.3/13B · 204B training tokens | overall performance | Helped |
| Increase EMA smoothing factor from 0.1 to 0.2 (ablation) | Seed-MoE-1.3/13B · 204B training tokens | overall performance | Helped |
| Use different merging methods (SMA, WMA, or EMA) (ablation) | Seed-MoE-1.3/13B · Later training stages | overall performance | No clear effect |
| Use a larger checkpoint interval, V=16B or V=32B, rather than smaller tested intervals (ablation) | Seed-MoE-1.3/13B · 204B training tokens | overall performance | Hurt |
| Increase the number of merged checkpoints from N=3 to N=15 (ablation) | Seed-MoE-1.3/13B · After training is complete | overall performance: nearly 1 point | Helped |
| Use more merged checkpoints (ablation) | Seed-MoE-1.3/13B · Early training | overall performance | Hurt |
| Initialize SFT from PMA weights instead of original weights, using the same learning-rate schedule (ablation) | Seed-MoE-15B/150B · OOD | score: 32.6 -> 34.7 | Helped |
| Initialize SFT from PMA weights and use a 1e-5 -> 2e-6 learning-rate schedule instead of the baseline schedule (ablation) | Seed-MoE-15B/150B · LiveBench | score: 50.5 -> 53.2 | Helped |
| Initialize downstream training from PMA weights (ablation) | PMA-init for CT and SFT · Downstream training | final performance | No clear effect |
Surprising findings
- Merging early in the cosine annealing phase often matches end-of-annealing performance, and sometimes surpasses it for larger models.
- Performance differences among SMA, WMA and EMA become negligible at later training stages.
Limitations
- Experiments rely on proprietary internal models and corpus, limiting reproducibility.
- Learning rate impact on merging was not systematically studied; default scaling-law LR used.
- Post-training (RL) merging not investigated; PMA-init SFT gains were not consistently replicated across model sizes.
Experimental setup
Datasets: internal pre-training corpus (trillions of tokens) in-house SFT data
Benchmarks: ARC-Challenge BBH DROP WinoGrande HellaSwag MMLU C-Eval TriviaQA Ape210K GSM8K MATH MBPP
Metrics: weighted average score accuracy
Baselines: cosine annealing final checkpoint constant learning rate checkpoint EMA WMA
Builds on: LAWA WSD learning rate schedule DARE Task Arithmetic
Closest papers by meaning
- Activation-Informed Merging of Large Language Models (NeurIPS 2025 · similarity 0.92)
- MergeBench: A Benchmark for Merging Domain-Specialized LLMs (NeurIPS 2025 · similarity 0.92)
- Channel Merging: Preserving Specialization for Merged Experts (AAAI 2025 · similarity 0.91)
- AdaMMS: Model Merging for Heterogeneous Multimodal Large Language Models with Unsupervised Coefficient Optimization (CVPR 2025 · similarity 0.91)
- $\texttt{Model-GLUE}$: Democratized LLM Scaling for A Large Model Zoo in the Wild (NeurIPS 2024 · similarity 0.91)
- OptMerge: Unifying Multimodal LLM Capabilities and Modalities via Model Merging (ICLR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.