Research map / Architectures and efficiency / RNNs and normalization
Midtraining Bridges Pretraining and Posttraining Distributions
Emmy Liu, Graham Neubig, Chenyan Xiong
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| midtrain on code (Starcoder 20%) before SFT | Pythia-1B · Pycode SFT | SFT validation loss: 2.174 -> 1.888 | Improved |
| midtrain on math (12%) before SFT | Pythia-1B · GSM8K SFT | SFT validation loss: 0.942 -> 0.851 | Improved |
| use mismatched midtraining domain (math for code, code for math) (ablation) | Pythia-1B · Pycode and GSM8K SFT | SFT validation loss: Pycode 2.174 -> 2.175 (Math mix); GSM8K 0.942 -> 0.927 (Starcoder mix) | No clear effect |
| midtrain on FLAN instruction data (ablation) | Pythia-1B · Pycode, GSM8K, LIMA, SciQ SFT | SFT validation loss: Pycode 2.174 -> 2.174; GSM8K 0.942 -> 0.942 | No clear effect |
| midtrain on code or math instead of C4 continuation, effect on C4 loss after SFT | Pythia-1B · GSM8K SFT | C4 validation loss after SFT: 3.134 -> 3.018 | Improved |
| matched midtraining reduces SFT-induced C4 degradation | Pythia-1B · Pycode and GSM8K | posttraining forgetting (C4 loss increase): Pycode 0.073 -> 0.038; GSM8K 0.132 -> 0.020 | Improved |
| choose midtraining data with larger proximity advantage to target | Pythia-1B · midtrain/SFT pairs | relative improvement vs proximity advantage: r=0.622, p=0.003 | Improved |
| keep general data in mixture rather than 100% specialized continued pretraining (code) (ablation) | Pythia-160M · Pycode SFT | SFT validation loss: 2.219 -> 2.134 | Improved |
| keep general data in mixture rather than 100% specialized continued pretraining (code), C4 retention (ablation) | Pythia-160M · Pycode SFT | C4 validation loss: 5.369 -> 5.079 | Improved |
| keep general data in mixture rather than 100% math continued pretraining (ablation) | Pythia-70M · GSM8K SFT | SFT validation loss: 1.383 -> 1.339 | Improved |
| raise mixture weight to 80% with early introduction (ablation) | Starcoder midtraining, 70M/160M · Pycode SFT | SFT validation loss | Improved |
| raise mixture weight to 80% with late introduction (105B tokens) (ablation) | Starcoder midtraining, 70M/160M · Pycode SFT | SFT validation loss | Hurt |
Closest papers by meaning
- Distilled Pretraining: A modern lens of Data, In-Context Learning and Test-Time Scaling (ICLR 2026 · similarity 0.89)
- Improving Pretraining Data Using Perplexity Correlations (ICLR 2025 · similarity 0.88)
- An Empirical Study on Noisy Data and LLM Pretraining Loss Divergence (NeurIPS 2026 · similarity 0.88)
- Scaling Laws for Forgetting during Finetuning with Pretraining Data Injection (ICML 2025 · similarity 0.88)
- Bridging Compute- and Data-Optimal Pretraining (NeurIPS 2026 · similarity 0.88)
- Tracing the Representation Geometry of Language Models from Pretraining to Post-training (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
