Research map / Large language models / Retrieval and knowledge
Propagating Knowledge Updates to LMs Through Distillation
Shankar Padmanabhan, Yasumasa Onoe, Michael JQ Zhang, Greg Durrett, Eunsol Choi
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use GPT-3.5 instead of the base model to generate the transfer set (ablation) | context distillation · Entity Inferences, GPT-Neo-1.3B | accuracy: 61.8 -> 65.9 | Improved |
| use GPT-3.5 instead of the base model to generate the transfer set (ablation) | context distillation · ECBD 2022, GPT-Neo-1.3B | target perplexity: 26.0 -> 25.3 | Improved |
| replace fine-tuning on the definition with distillation | GPT-Neo-1.3B · ECBD 2022 | target perplexity: 28.5 -> 25.3 | Improved |
| also fine-tune on the transfer set in addition to the definition (ablation) | fine-tuning · ECBD 2022, GPT-Neo-1.3B | target perplexity: 28.5 -> 28.9 | Hurt |
| also fine-tune on the transfer set in addition to the definition (ablation) | fine-tuning · Entity Inferences, GPT-Neo-1.3B | accuracy: 57.7 -> 66.5 | Improved |
| use a random entity's definition instead of the correct one (ablation) | distillation · ECBD 2022, GPT-Neo-1.3B | target perplexity: 25.3 -> 33.6 | Hurt |
| use a random transfer set (correct definition) (ablation) | distillation · ECBD 2022, GPT-Neo-1.3B | target perplexity: 25.3 -> 28.9 | Hurt |
| prepend entity name to random transfer set elements (ablation) | distillation · ECBD 2022, GPT-Neo-1.3B | target perplexity: 28.9 -> 26.7 | Improved |
| prepend a random definition to the context | GPT-Neo-1.3B · ECBD 2022 | target perplexity: 31.0 -> 42.9 | Hurt |
| use ten unique continuations instead of one repeated ten times | distillation · ECBD 2022, GPT-Neo-1.3B | target perplexity: 25.3 -> 23.5 | Improved |
| apply distillation to popular entities instead of fine-tuning | GPT-Neo-1.3B · ECBD Popular | target perplexity: 36.6 -> 34.5 | Improved |
| use distillation instead of ROME for counterfactual edits | GPT2-XL · CounterFact | Score: 32.3 -> 42.2 | Improved |
Closest papers by meaning
- DistiLLM: Towards Streamlined Distillation for Large Language Models (ICML 2024 · similarity 0.88)
- From Static to Dynamic: Knowledge Metabolism for Large Language Models (AAAI 2024 · similarity 0.88)
- WASD: Wasserstein-based Knowledge Distillation for Large Language Models (NeurIPS 2026 · similarity 0.88)
- DDK: Distilling Domain Knowledge for Efficient Large Language Models (NeurIPS 2024 · similarity 0.87)
- MiniPLM: Knowledge Distillation for Pre-training Language Models (ICLR 2025 · similarity 0.87)
- Why Knowledge Distillation Works in Generative Models: A Minimal Working Explanation (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
