AI Paper Atlas

Research map / Architectures and efficiency / RNNs and normalization

Midtraining Bridges Pretraining and Posttraining Distributions

Emmy Liu, Graham Neubig, Chenyan Xiong

ICML 2026 · Architectures and efficiency › RNNs and normalization · approach: Layer-wise analysis of language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
midtrain on code (Starcoder 20%) before SFTPythia-1B · Pycode SFTSFT validation loss: 2.174 -> 1.888Improved
midtrain on math (12%) before SFTPythia-1B · GSM8K SFTSFT validation loss: 0.942 -> 0.851Improved
use mismatched midtraining domain (math for code, code for math) (ablation)Pythia-1B · Pycode and GSM8K SFTSFT validation loss: Pycode 2.174 -> 2.175 (Math mix); GSM8K 0.942 -> 0.927 (Starcoder mix)No clear effect
midtrain on FLAN instruction data (ablation)Pythia-1B · Pycode, GSM8K, LIMA, SciQ SFTSFT validation loss: Pycode 2.174 -> 2.174; GSM8K 0.942 -> 0.942No clear effect
midtrain on code or math instead of C4 continuation, effect on C4 loss after SFTPythia-1B · GSM8K SFTC4 validation loss after SFT: 3.134 -> 3.018Improved
matched midtraining reduces SFT-induced C4 degradationPythia-1B · Pycode and GSM8Kposttraining forgetting (C4 loss increase): Pycode 0.073 -> 0.038; GSM8K 0.132 -> 0.020Improved
choose midtraining data with larger proximity advantage to targetPythia-1B · midtrain/SFT pairsrelative improvement vs proximity advantage: r=0.622, p=0.003Improved
keep general data in mixture rather than 100% specialized continued pretraining (code) (ablation)Pythia-160M · Pycode SFTSFT validation loss: 2.219 -> 2.134Improved
keep general data in mixture rather than 100% specialized continued pretraining (code), C4 retention (ablation)Pythia-160M · Pycode SFTC4 validation loss: 5.369 -> 5.079Improved
keep general data in mixture rather than 100% math continued pretraining (ablation)Pythia-70M · GSM8K SFTSFT validation loss: 1.383 -> 1.339Improved
raise mixture weight to 80% with early introduction (ablation)Starcoder midtraining, 70M/160M · Pycode SFTSFT validation lossImproved
raise mixture weight to 80% with late introduction (105B tokens) (ablation)Starcoder midtraining, 70M/160M · Pycode SFTSFT validation lossHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy