Research map / Large language models / Reasoning and chain of thought

Distilled Pretraining: A modern lens of Data, In-Context Learning and Test-Time Scaling

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Routing, pruning and distillation for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Studies distilled LLM pretraining (soft-label distillation) and finds it improves test-time scaling via pass@k diversity while impairing in-context learning via induction heads, explained by a bigram sandbox and a generalized pass@k Bayes-optimal analysis.

Problem

Whether the benefits of distilled pretraining hold when student and teacher see the same data, and how it affects modern capabilities like in-context learning and test-time scaling (pass@k).

Approach

Shows distilled pretraining trades off ICL (weakened induction heads, since soft labels on low-entropy deterministic transitions add noise) for better pass@k diversity (learning high-entropy rows faster), explained via a bigram sandbox and sample-complexity and generalized pass@k Bayes-optimality analyses; proposes entropy-based token routing to mitigate the ICL drop.

Key results

• On GSM8K, distilled pretraining (DPT-50) reaches pass@16 of 28% vs 23% for standard pretraining with similar pass@1.
• DPT-90 trained on half the data exceeds SPT-2x pass@16 on GSM8K, MATH and MBPP despite lower pass@1.
• In the IsoData setting, distillation underperforms standard pretraining on needle-in-haystack and counterfactual QA ICL tasks; token routing (skipping distillation on 15% lowest-entropy tokens) partially closes this gap.

Surprising findings

• Distillation with an IsoData teacher hurts in-context learning even as it improves standard benchmarks and pass@k.
• An RL-trained teacher yields better students than a base teacher, even on general benchmarks like HellaSwag and TQA.

Limitations

• Experiments are on 1B students with 125B-1T tokens; scale generality unverified.
• Diversity studied only via pass@k on reasoning and coding; other diversity notions (creativity) out of scope.
• Token routing is preliminary and helps on only two of three ICL tasks.

Experimental setup

Datasets: DCLM GitHub repositories DeepMind Mathematics Proof Pile 2 Stack Exchange (the Pile) FineWeb-Edu Natural Reasoning AQuA

Benchmarks: GSM8K MATH500 MBPP HellaSwag TQA HumanEval+ GPQA ARC-C ARC-E COPA SQuAD DROP

Metrics: pass@k pass@1 accuracy F1 EM KL divergence

Baselines: standard next-token pretraining (SPT) SPT-2x multi-token prediction (MTP) Llama-3.1-8B-based teacher variants

Builds on: knowledge distillation (Hinton et al.) induction heads multi-token prediction DCLM

Closest papers by meaning

• Strong Teacher Not Needed? On Distillation in LLM Pretraining (NeurIPS 2026 · similarity 0.92)
• DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs (ICML 2025 · similarity 0.91)
• DistiLLM: Towards Streamlined Distillation for Large Language Models (ICML 2024 · similarity 0.91)
• Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models (NeurIPS 2026 · similarity 0.90)
• MiniPLM: Knowledge Distillation for Pre-training Language Models (ICLR 2025 · similarity 0.90)
• Why Knowledge Distillation Works in Generative Models: A Minimal Working Explanation (NeurIPS 2025 · similarity 0.89)