Research map / Label-efficient and robust learning / Knowledge distillation
Induced Model Matching: Restricted Models Help Train Full-Featured Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Induced Model Matching (IMM), which regularizes a full-featured model by matching its context-restricted (induced) predictions to a restricted target model, and shows it is consistent unlike noising and reverse knowledge distillation.
Problem
How to use a very accurate feature-restricted model (e.g., bigram LM or single-feature predictor) to improve training of a full-featured model with a larger feature set, given that noising and reverse-KD may be inconsistent.
Approach
Match the induced version of the learned full model (averaging its predictions over extended contexts with the short context fixed) to a target restricted model via a cross-entropy regularizer, instead of matching the full model directly as in noising or reverse-KD; practical via sampled and serialized gradient approximations.
Key results
- On PTB LSTM LM, IMM with KN bigram reaches 73.3 test perplexity versus 77.5 for no-IMM baseline and 73.9 for KN noising.
- On BERT-BASE fine-tuning, +IMM improves CoLA to 60.0 (from 52.1 baseline) and RTE to 71 (from 66).
- In logistic regression, IMM always improves on baseline and reduces variance; the gain is equivalent to about 30% extra data on average.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add sampled IMM with a Kneser-Ney bigram | LSTM language model · Penn TreeBank | validation perplexity: 81.6 -> 76.0 | Helped |
| Add sampled IMM with a Kneser-Ney bigram | LSTM language model · Penn TreeBank | test perplexity: 77.5 -> 73.3 | Helped |
| Replace Kneser-Ney bigram noising with IMM | LSTM language model · Penn TreeBank | validation perplexity: 76.7 -> 76.0 | Helped |
| Replace Kneser-Ney bigram noising with IMM | LSTM language model · Penn TreeBank | test perplexity: 73.9 -> 73.3 | Helped |
| Reintroduce the MLM objective during fine-tuning (ablation) | BERT_BASE · CoLA | Matthew's Correlation Coefficient: 52.1 ± 4.0 -> 55.0 ± 3.0 | Helped |
| Add IMM to MLM during fine-tuning | BERT_BASE · CoLA | Matthew's Correlation Coefficient: 55.0 ± 3.0 -> 60.0 ± 1.0 | Helped |
| Reintroduce the MLM objective during fine-tuning (ablation) | BERT_BASE · MRPC | F1: 88.9 ± 2.0 -> 89 ± 1.0 | No clear effect |
| Add IMM to MLM during fine-tuning | BERT_BASE · MRPC | F1: 89 ± 1.0 -> 90 ± 1.0 | Helped |
| Reintroduce the MLM objective during fine-tuning (ablation) | BERT_BASE · QNLI | accuracy: 90.5 ± 2.0 -> 91.0 ± 2.0 | No clear effect |
| Add IMM to MLM during fine-tuning | BERT_BASE · QNLI | accuracy: 91.0 ± 2.0 -> 93.5 ± 1.0 | Helped |
| Reintroduce the MLM objective during fine-tuning (ablation) | BERT_BASE · RTE | accuracy: 66 ± 3.0 -> 68 ± 2.0 | Helped |
| Add IMM to MLM during fine-tuning | BERT_BASE · RTE | accuracy: 68 ± 2.0 -> 71 ± 1.0 | Helped |
Surprising findings
- Noising and reverse-KD can be inconsistent: a counterexample shows the noising objective's minimizer can differ from the true model even in the realizable infinite-data case.
- With a fixed lambda, noising underperforms the baseline in logistic regression as data grows, while IMM remains stable.
Limitations
- Exact induced model computation is not viable, requiring sampling approximations with k-fold time overhead.
- Sampled IMM overhead can be O(kL) for LSTMs unless applied periodically.
- Experiments are small-scale proof-of-concept (PTB LSTM, BERT-BASE on subset of GLUE, toy RL).
Experimental setup
Datasets: Penn Treebank GLUE CoLA GLUE MRPC GLUE QNLI GLUE RTE synthetic logistic regression data
Benchmarks: Penn Treebank GLUE CoLA GLUE MRPC GLUE QNLI GLUE RTE
Metrics: perplexity Matthews correlation F1 accuracy average reward
Baselines: no-IMM baseline KN noising interpolation noising MLM
Builds on: knowledge distillation noising data augmentation reverse knowledge distillation Kneser-Ney smoothing REINFORCE
Closest papers by meaning
- In-Context Unlearning: Language Models as Few-Shot Unlearners (ICML 2024 · similarity 0.84)
- Cross-model Control: Improving Multiple Large Language Models in One-time Training (NeurIPS 2024 · similarity 0.84)
- Scaling up Masked Diffusion Models on Text (ICLR 2025 · similarity 0.84)
- Customizing Language Model Responses with Contrastive In-Context Learning (AAAI 2024 · similarity 0.84)
- Faithful Explanations of Black-box NLP Models Using LLM-generated Counterfactuals (ICLR 2024 · similarity 0.84)
- FlexOLMo: Open Language Models for Flexible Data Use (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.