AI Paper Atlas

Research map / Large language models / Retrieval and knowledge

Propagating Knowledge Updates to LMs Through Distillation

Shankar Padmanabhan, Yasumasa Onoe, Michael JQ Zhang, Greg Durrett, Eunsol Choi

NeurIPS 2023 · Large language models › Retrieval and knowledge · approach: Knowledge editing in LLMs · 4 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use GPT-3.5 instead of the base model to generate the transfer set (ablation)context distillation · Entity Inferences, GPT-Neo-1.3Baccuracy: 61.8 -> 65.9Improved
use GPT-3.5 instead of the base model to generate the transfer set (ablation)context distillation · ECBD 2022, GPT-Neo-1.3Btarget perplexity: 26.0 -> 25.3Improved
replace fine-tuning on the definition with distillationGPT-Neo-1.3B · ECBD 2022target perplexity: 28.5 -> 25.3Improved
also fine-tune on the transfer set in addition to the definition (ablation)fine-tuning · ECBD 2022, GPT-Neo-1.3Btarget perplexity: 28.5 -> 28.9Hurt
also fine-tune on the transfer set in addition to the definition (ablation)fine-tuning · Entity Inferences, GPT-Neo-1.3Baccuracy: 57.7 -> 66.5Improved
use a random entity's definition instead of the correct one (ablation)distillation · ECBD 2022, GPT-Neo-1.3Btarget perplexity: 25.3 -> 33.6Hurt
use a random transfer set (correct definition) (ablation)distillation · ECBD 2022, GPT-Neo-1.3Btarget perplexity: 25.3 -> 28.9Hurt
prepend entity name to random transfer set elements (ablation)distillation · ECBD 2022, GPT-Neo-1.3Btarget perplexity: 28.9 -> 26.7Improved
prepend a random definition to the contextGPT-Neo-1.3B · ECBD 2022target perplexity: 31.0 -> 42.9Hurt
use ten unique continuations instead of one repeated ten timesdistillation · ECBD 2022, GPT-Neo-1.3Btarget perplexity: 25.3 -> 23.5Improved
apply distillation to popular entities instead of fine-tuningGPT-Neo-1.3B · ECBD Populartarget perplexity: 36.6 -> 34.5Improved
use distillation instead of ROME for counterfactual editsGPT2-XL · CounterFactScore: 32.3 -> 42.2Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy