Research map / Label-efficient and robust learning / Knowledge distillation

Strong Teacher Not Needed? On Distillation in LLM Pretraining

NeurIPS 2026 · Label-efficient and robust learning › Knowledge distillation · approach: Rethinking distillation objectives · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Systematically studies logit-level knowledge distillation during LLM pretraining across teacher-student pairs varying in architecture size and training tokens, finding weak teachers can help and stronger teachers can saturate or reverse gains.

Problem

Knowledge distillation is assumed to require a strong-to-weak teacher-student relationship where stronger teachers yield better students; this has not been systematically tested for LLM pretraining across architecture sizes and token budgets.

Approach

Vary teacher architecture size (0.7B to 8.0B) and training tokens (10B to 300B) against a fixed student budget to create strong-to-weak, same-level, and weak-to-strong relationships, and tune the mixing coefficient between LM and forward-KL distillation losses to measure distillation gains.

Key results

• Even teachers weaker than the student help: the 0.7B teacher at 30B tokens yields +1.3% downstream accuracy for a 1.7B student trained on 50B tokens.
• At 300B tokens, the 1.7B teacher gives +4.3% downstream accuracy vs +3.5% for 3.8B and +2.8% for 8.0B teachers, so stronger teachers can underperform.
• The 8.0B teacher's downstream gain peaks at 30B tokens (+4.3%) and declines to +2.8% at 300B tokens.

Surprising findings

• Same-level distillation (same architecture and tokens, different seed) improves the student across all metrics (+1.0% in-domain ppl, +3.5% OOD ppl, +1.5% downstream acc).
• Teacher train loss is a misleading proxy: smaller teachers with higher train loss can yield better OOD and downstream improvements than larger teachers with lower loss.

Limitations

• Single model family (Llama3-style architectures) and a single pretraining corpus (FineWeb-Edu).
• Student fixed at 50B tokens in main experiments; gains measured relative to a single baseline per student size.
• Downstream accuracy is noisier than perplexity trends, and improvements are small in absolute terms (often a few percent).

Experimental setup

Datasets: FineWeb-Edu Wikitext-103 C4 GSM8K DM Mathematics HumanEval CodeSearchNet arXiv CNN-DailyMail ECHR PubMedQA XQuAD

Benchmarks: FineWeb-Edu held-out perplexity 11 OOD corpora perplexity MMLU ARC-Easy SciQ OpenBookQA MathQA TruthfulQA ANLI-R1 CommonsenseQA HellaSwag PIQA

Metrics: perplexity accuracy percentage improvement over baseline

Baselines: standard LM pretraining baseline (alpha=0) pure distillation (alpha=1.0)

Builds on: knowledge distillation (Hinton et al.) Born-Again Networks Llama 3

Closest papers by meaning

• Distilled Pretraining: A modern lens of Data, In-Context Learning and Test-Time Scaling (ICLR 2026 · similarity 0.92)
• DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs (ICML 2025 · similarity 0.91)
• Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models (NeurIPS 2026 · similarity 0.91)
• On Teacher Hacking in Language Model Distillation (ICML 2025 · similarity 0.89)
• Progressive distillation induces an implicit curriculum (ICLR 2025 · similarity 0.89)
• MiniPLM: Knowledge Distillation for Pre-training Language Models (ICLR 2025 · similarity 0.89)