Research map / Large language models / Alignment and preferences
On Teacher Hacking in Language Model Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces "teacher hacking" in LM distillation and tests it in a semi-synthetic setup with a Flan-T5-XL oracle, T5-1.1 teachers and students, finding that fixed offline data leads to hacking while online data or prompt diversity mitigates it.
Problem
Knowledge distillation treats the teacher as the target, but the teacher is only an imperfect proxy for the ground-truth distribution; the student may over-optimize toward the teacher's imperfections, analogous to reward hacking in RLHF.
Approach
Define teacher hacking and build a controlled oracle-teacher-student setup where the oracle provides golden metrics, so that the proxy (student-teacher distance) and golden (student-oracle distance) curves can be compared; show that offline distillation over many epochs hacks, detectable as deviation from polynomial convergence in the proxy metric, while online data sampling and data diversity mit
Key results
- On XSum with T5-large teacher and T5-base student, the golden metric first decreases and then increases under offline distillation (U-shaped proxy-golden curve), while the proxy metric keeps decreasing.
- Online teacher or student data sampling avoids teacher hacking; mixing as little as 10% online student data stabilizes the golden metric.
- Reducing offline prompt diversity (k=2 or k=5 generations per prompt at fixed budget) makes teacher hacking more evident on XSum and Natural Instructions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train on a fixed offline teacher-generated dataset for extended distillation | T5-large to T5-base distillation · XSum | golden metric (oracle-student distance) | Hurt |
| Train on a fixed offline teacher-generated dataset for extended distillation | T5-large to T5-base distillation · XSum | proxy metric (teacher-student distance) | Helped |
| Generate responses online from the teacher instead of using a fixed dataset | T5-large to T5-base distillation · XSum | golden metric (oracle-student distance) | Helped |
| Generate responses online from the student instead of using a fixed dataset | T5-large to T5-base distillation · XSum | golden metric (oracle-student distance) | Helped |
| Use fewer prompts and more generations per prompt at a fixed dataset size (ablation) | Offline T5-large to T5-base distillation · XSum | golden metric (oracle-student distance) | Hurt |
| Reduce prompt diversity at a fixed generation budget (ablation) | Offline T5-large to T5-base distillation · Natural Instructions | golden metric (oracle-student distance) | Hurt |
| Reduce prompt diversity at a fixed generation budget (ablation) | Offline T5-large to T5-base distillation · WMT-14 en-de | golden metric (oracle-student distance) | No clear effect |
| Increase the generation budget with multiple completions per prompt (ablation) | Offline T5-large to T5-base distillation · XSum | golden metric (oracle-student distance) | Helped |
| Add online student-generated data to offline data (ablation) | T5-large to T5-base distillation · XSum | golden metric (oracle-student distance) | Helped |
| Replace forward KL with reverse KL or Jensen-Shannon divergence (ablation) | Offline T5-large to T5-base distillation · XSum | golden metric (oracle-student distance) | Hurt |
| Use generalized Jensen-Shannon divergence with coefficient 0.1 or 0.9 (ablation) | T5-large to T5-base distillation · XSum | golden metric (oracle-student distance) | Mixed |
| Limit offline distillation to a few epochs (ablation) | T5-large to T5-base distillation · XSum | golden metric (oracle-student distance) | No clear effect |
Surprising findings
- Distilling T5-large into T5-small showed standard overfitting (proxy metric slightly increased) rather than teacher hacking.
- Diversity reduction had almost no effect on WMT-14 en-de, attributed to its initially low diversity and short sentences.
Limitations
- Semi-synthetic setup: the oracle is a Flan-T5-XL model rather than true human data, and all models are small T5-family models.
- Evaluation is confined to a controlled setup with distances estimated on held-out prompts; generalization to large-scale post-training is not tested.
- Only soft distillation with a few token-level losses is studied; generalized JS variants were excluded as not competitive.
Experimental setup
Datasets: XSum WMT-14 en-de Natural Instructions
Benchmarks: XSum WMT-14 en-de Natural Instructions
Metrics: sequence-level forward KL (golden, to oracle) sequence-level reverse KL sequence-level Jensen-Shannon distance proxy KL to teacher
Baselines: SFT online teacher data source online student data source reverse KL loss Jensen-Shannon divergence loss
Builds on: knowledge distillation sequence-level KD reward hacking analysis (gao et al. 2023 style setup) T5
Closest papers by meaning
- Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models (NeurIPS 2026 · similarity 0.90)
- Strong Teacher Not Needed? On Distillation in LLM Pretraining (NeurIPS 2026 · similarity 0.89)
- Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use (ICML 2026 · similarity 0.89)
- Inference-Time Reward Hacking in Large Language Models (NeurIPS 2025 · similarity 0.88)
- Why Knowledge Distillation Works in Generative Models: A Minimal Working Explanation (NeurIPS 2025 · similarity 0.88)
- IFDECORATOR: Wrapping Instruction Following Reinforcement Learning with Verifiable Rewards (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.