Research map / Large language models / Instruction and fine-tuning
MEND: Meta Demonstration Distillation for Efficient and Effective In-Context Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MEND, a small language model that distills lengthy ICL demonstrations into compact vectors, trained with a KL-based knowledge distillation objective through a frozen LLM in meta-distillation pretraining (C4) and finetuning (MetaICL).
Problem
In-context learning with many demonstrations incurs quadratic self-attention cost; existing demonstration distillation into vectors (hypernetworks, HyperTuning) either needs retraining per task or degrades ICL performance relative to vanilla ICL.
Approach
A small distillation LM (gpt2-small or t5-small-lm-adapt) produces soft prompt vectors from demonstrations and is trained so that a frozen LLM conditioned on these vectors matches the output distribution of the LLM conditioned on full demonstrations (KL distillation), after a meta-distillation pretraining stage on C4 and a finetuning stage on ICL tasks.
Key results
- MEND averages 43.05 on gpt2-large versus 41.93 for Vanilla ICL and 41.15 for HyperTuning on MetaICL partitions.
- MEND reaches 39.79 average on t5-lm-large versus 38.15 for Vanilla ICL, while HyperTuning averages 35.09.
- The paper reports up to 75% FLOPs reduction and up to 33% faster inference relative to vanilla ICL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MEND instead of Vanilla ICL | GPT-2 large · MetaICL AVG | score: 41.93 -> 43.05 | Helped |
| use MEND instead of Vanilla ICL | GPT-2 XL · MetaICL AVG | score: 42.15 -> 43.12 | Helped |
| use MEND instead of Vanilla ICL | T5 large · MetaICL AVG | score: 38.15 -> 39.79 | Helped |
| remove prediction loss during finetuning (ablation) | MEND · MetaICL average ablation | score: 43.65 -> 40.41 | Hurt |
| remove distillation loss during finetuning (ablation) | MEND · MetaICL average ablation | score: 43.65 -> 40.64 | Hurt |
| remove meta-distillation pretraining (ablation) | MEND · MetaICL average ablation | score: 43.65 -> 40.10 | Hurt |
| replace MEND pretraining objective with CLM (ablation) | MEND · MetaICL average ablation | score: 43.65 -> 42.92 | Hurt |
| combine CLM and distillation loss during pretraining (ablation) | MEND · MetaICL average ablation | score: 43.65 -> 42.24 | Hurt |
| increase the number of demonstrations from 1 through 16 (ablation) | MEND · Class → Class | score | Helped |
| increase distilled-vector length from 1 to 100 (ablation) | MEND · Class → Class | score | Helped |
| retrieve demonstrations with dense k-NN rather than random sampling (ablation) | MEND · Class → Class | score: 43.35 -> 48.81 | Helped |
| remove demonstration inputs while retaining labels (ablation) | MEND · Class → Class | score: 43.35 -> 44.29 | Helped |
Surprising findings
- HyperTuning underperforms Vanilla ICL on the T5 backbone and on the newline-template variant, while MEND stays robust.
- Pretraining with plain conditional language modeling plus distillation loss gave slightly worse results than distillation loss alone in the ablation.
Limitations
- Experiments limited to models under 2B parameters; generalization to 10B+ models untested.
- Distillation model must be retrained for each new LLM and backprop through the LLM adds overhead.
- Limited context window restricts the number of demonstrations that can be distilled.
Experimental setup
Datasets: C4 MetaICL CrossFit UnifiedQA
Benchmarks: MetaICL seven partitions (Class→Class, non-Class→Class, non-NLI→NLI, non-QA→QA, QA→QA, HR→LR, non-Para→Para)
Metrics: accuracy Macro-F1 FLOPs inference time peak GPU memory
Baselines: Zero-shot Vanilla ICL PromptTuning HyperTuning
Builds on: HyperTuning MetaICL Prompt Tuning Knowledge distillation
Closest papers by meaning
- Are Human-generated Demonstrations Necessary for In-context Learning? (ICLR 2024 · similarity 0.90)
- In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering (ICML 2024 · similarity 0.89)
- Mixture of Demonstrations for In-Context Learning (NeurIPS 2024 · similarity 0.89)
- DETAIL: Task DEmonsTration Attribution for Interpretable In-context Learning (NeurIPS 2024 · similarity 0.88)
- Comparative Analysis of Demonstration Selection Algorithms for In-Context Learning in Large Language Models (Student Abstract) (AAAI 2025 · similarity 0.88)
- TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.