Research map / Large language models / Language, safety and interpretability
Mixture of Demonstrations for In-Context Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MoD, a mixture-of-experts demonstration retriever for in-context learning that clusters the demonstration pool into expert groups and trains each expert with a coordinate-descent-inspired contrastive objective.
Problem
Demonstration selection for in-context learning suffers from a huge combinatorial search space over demonstration sets and from noisy training signals when unhelpful demonstrations are sampled alongside the candidate being scored.
Approach
Partition the demonstration pool via K-means into expert groups, train a retriever per expert, and use expert-wise (coordinate-descent-style) training that fixes the other selected demonstrations while scoring one candidate, so the supervision signal comes from a set of already-good demonstrations; at inference, experts jointly contribute demonstrations in proportion to query-centroid similarity.
Key results
- MoD achieves an average score of 58.63% across 12 tasks, versus 56.76% for CEIL and 53.37% for EPR.
- On GeoQuery Standard split, MoD reaches 77.38 vs. 73.21 for CEIL; on MRPC MoD gets 81.53 vs. CEIL 80.15.
- Ablation shows removing the mixture-of-experts design causes the largest drop, especially on classification tasks such as SST-5 and CMSQA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MoD instead of CEIL | ICL retriever with GPT-Neo · CMSQA validation | accuracy: 37.18 -> 43.24 | Helped |
| use MoD instead of CEIL | ICL retriever with GPT-Neo · SST-5 validation | accuracy: 47.05 -> 48.12 | Helped |
| use MoD instead of CEIL | ICL retriever with GPT-Neo · GeoQuery validation | exact match: 73.21 -> 73.75 | Helped |
| use MoD instead of CEIL | ICL retriever with GPT-Neo · MTOP validation | exact match: 67.43 -> 69.32 | Helped |
| add few-shot scoring (ablation) | MoD · SST-5 | accuracy: 44.07 -> 48.12 | Helped |
| add few-shot scoring (ablation) | MoD · CMSQA | accuracy: 41.65 -> 43.24 | Helped |
| add few-shot scoring (ablation) | MoD · GeoQuery | exact match: 71.35 -> 73.75 | Helped |
| add few-shot scoring (ablation) | MoD · MTOP | exact match: 64.36 -> 69.32 | Helped |
| select the highest-scored fixed demonstrations instead of random ones (ablation) | MoD · SST-5 | accuracy: 46.42 -> 48.12 | Helped |
| select the highest-scored fixed demonstrations instead of random ones (ablation) | MoD · CMSQA | accuracy: 42.69 -> 43.24 | Helped |
| select the highest-scored fixed demonstrations instead of random ones (ablation) | MoD · GeoQuery | exact match: 72.77 -> 73.75 | Helped |
| select the highest-scored fixed demonstrations instead of random ones (ablation) | MoD · MTOP | exact match: 66.89 -> 69.32 | Helped |
Surprising findings
- Retrievers trained on GPT-Neo transfer to GPT-3.5 with gains over TopK-BERT, though gains shrink for stronger models.
- Retriever transfer across tasks is generally harmful, with MNLI-trained retrievers losing up to 31.91 points on MTOP.
Limitations
- Requires labeled demonstrations to supply LLM-based supervision.
- Performance depends on correct expert assignment; misassigned queries may retrieve unhelpful demonstrations.
- Evaluated only on validation sets with GPT-Neo-scale models as the primary backbone.
Experimental setup
Datasets: SST-5 MRPC MNLI QNLI CMSQA HellaSwag WebQs GeoQuery NL2Bash Break MTOP SMCalFlow
Benchmarks: SST-5 MRPC QNLI MNLI CMSQA HellaSwag WebQs GeoQuery NL2Bash Break MTOP SMCalFlow
Metrics: accuracy exact match LF-EM
Baselines: Random TopK-BM25 TopK-Contriver TopK-SimCSE TopK-BERT EPR CEIL
Builds on: CEIL EPR mixture of experts contrastive learning
Closest papers by meaning
- What Factors Affect Multi-Modal In-Context Learning? An In-Depth Exploration (NeurIPS 2024 · similarity 0.90)
- Comparative Analysis of Demonstration Selection Algorithms for In-Context Learning in Large Language Models (Student Abstract) (AAAI 2025 · similarity 0.90)
- Revisiting In-context Learning Inference Circuit in Large Language Models (ICLR 2025 · similarity 0.90)
- Large Language Models are Demonstration Pre-Selectors for Themselves (ICML 2025 · similarity 0.89)
- DETAIL: Task DEmonsTration Attribution for Interpretable In-context Learning (NeurIPS 2024 · similarity 0.89)
- MEND: Meta Demonstration Distillation for Efficient and Effective In-Context Learning (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.