Research map / Label-efficient and robust learning / Recognition and long-tail
Learning from Teaching Regularization: Generalizable Correlations Should be Easy to Imitate
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Learning from Teaching (LoT), a regularizer where a teacher model is penalized by how poorly student models imitate it (KL divergence), improving generalization across RL, language modeling, and image classification.
Problem
Improving generalization of deep neural networks by identifying generalizable (simple, imitable) correlations rather than spurious ones.
Approach
Treat the imitability of the main model by auxiliary student learners as a regularizer: the teacher's objective adds a KL-divergence term between student and teacher predictions, so the teacher is pushed toward correlations that students can easily imitate, which the authors hypothesize are generalizable.
Key results
- On PTB with LSTM, LoT reduces test perplexity from 82.75 to 71.72 versus Teacher-only.
- On GSM8K with LLaMA-2 7B, LoT improves SFT accuracy from 39.81 to 41.87.
- On CIFAR-100 with ResNet-18 teacher and ResNet-50 student, LoT improves accuracy from 81.14 to 83.13 (ImageNet-1K pretrained).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add LoT regularization | LSTM and AWD-LSTM language models · PTB test | perplexity: 82.75 ± 0.36 -> 71.72 ± 0.54; 58.69 ± 0.37 -> 53.31 ± 0.56 | Helped |
| Add LoT regularization | Transformer-XL-B and Transformer-XL-L · WikiText-103 test | perplexity: 23.72 ± 0.41 -> 21.65 ± 0.38; 18.50 ± 0.25 -> 16.47 ± 0.23 | Helped |
| Use LoT during supervised fine-tuning instead of SFT alone | LLaMA-1 7B · GSM8K and MATH | accuracy: 34.39 ± 1.28 -> 36.42 ± 1.46; 4.78 ± 0.23 -> 5.39 ± 0.28 | Helped |
| Use LoT during supervised fine-tuning instead of SFT alone | LLaMA-2 7B · GSM8K and MATH | accuracy: 39.81 ± 1.34 -> 41.87 ± 1.62; 5.79 ± 0.31 -> 6.28 ± 0.22 | Helped |
| Add LoT with a ResNet-50 student | ResNet-18 teacher · CIFAR-100 | accuracy: 81.14 ± 0.58 -> 83.13 ± 0.26 | Helped |
| Add LoT with a ViT-L/16 student | ViT-B/16 teacher · ImageNet-1K | accuracy: 83.97 ± 0.11 -> 84.80 ± 0.08 | Helped |
| Add LoT to the PPO agent | Teacher-only Atari agents · BeamRider, Gravitar, UpNDown, Breakout | normalized return enhancement: 63.14%; 9.79%; 66.48%; 35.70% | Helped |
| Replace KL-divergence imitability metric with L2 loss (ablation) | LoT ViT-B/16 teacher and student · CIFAR-100 | accuracy: 93.17 -> 92.77 | Hurt |
| Increase the regularization coefficient from 0 to 1 (ablation) | LoT LSTM teacher and student · PTB test | perplexity | Helped |
| Increase the regularization coefficient above 1 (ablation) | LoT LSTM teacher and student · PTB test | perplexity | Hurt |
| Use a moderate student steps ratio such as 4 or 5 (ablation) | LoT LSTM teacher and student · PTB test | perplexity | Helped |
| Increase student steps ratio too far (ablation) | LoT LSTM teacher and student · PTB test | perplexity | Hurt |
Surprising findings
- Using a smaller student (e.g., MobileNetV2) still improves a larger teacher's accuracy.
- A mutual-imitation regularizer with a KL metric outperforms L2 imitability metric and distillation baselines BAN, ReviewKD and DKD on CIFAR-100.
Limitations
- Requires additional computation and memory for student models.
- Hyperparameter sensitivity: alpha and student steps ratio N need tuning, with performance declining for large alpha.
- Hypothesis about generalizable correlations being easier to imitate is validated only on a constructed sophisticated vs. deceptive teacher setup.
Experimental setup
Datasets: CIFAR-100 ImageNet-1K ImageNet-21K Penn Tree Bank WikiText-103 GSM8K MATH BeamRider Breakout UpNDown Gravitar ImageNet-R
Benchmarks: CIFAR-100 ImageNet-1K PTB WikiText-103 GSM8K MATH Atari BeamRider Atari Breakout Atari UpNDown Atari Gravitar ImageNet-R ImageNet-Sketch
Metrics: test accuracy perplexity episodic return normalized return enhancement KL divergence
Baselines: Teacher-only BAN ReviewKD DKD ICL SFT
Builds on: knowledge distillation born-again networks PPO iterated learning
Closest papers by meaning
- Intraclass clustering: an implicit learning ability that regularizes DNNs (ICLR 2021 · similarity 0.87)
- Weak-to-Strong Generalization Even in Random Feature Networks, Provably (ICML 2025 · similarity 0.87)
- On the Mechanisms of Weak-to-Strong Generalization: A Theoretical Perspective (NeurIPS 2025 · similarity 0.87)
- To Grok or not to Grok: Disentangling Generalization and Memorization on Corrupted Algorithmic Datasets (ICLR 2024 · similarity 0.86)
- Explaining Generalization Power of a DNN Using Interactive Concepts (AAAI 2024 · similarity 0.85)
- From Linear to Nonlinear: Provable Weak-to-Strong Generalization through Feature Learning (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.