Research map / Large language models / Language, safety and interpretability
Compositional Exemplars for In-context Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Learning-based DPP retriever trained with contrastive pairwise loss (CEIL) vs. EPR singleton retriever | GPT-Neo 2.7B in-context selection · 12 datasets average | Avg. score: +3.39 | Helped |
| Initialize the CEIL encoder from EPR rather than raw BERT (ablation) | CEIL training · SST-5, MRPC, QNLI, GeoQuery, MTOP | Accuracy/EM: SST5 31.34 to 49.14 (InfoNCE); MTOP 47.43 to 61.92 (InfoNCE) | Helped |
| Pair-wise margin loss instead of InfoNCE (ablation) | CEIL training with EPR init · GeoQuery and MTOP | EM: GeoQuery 69.29 to 73.21 (+3.92); MTOP 61.92 to 67.43 (+5.51); SST5 49.14 to 47.05 (-2.09) | Mixed |
| Pair-wise margin loss instead of InfoNCE (ablation) | CEIL training with EPR init · classification tasks SST-5, MRPC, QNLI | Accuracy: SST5 49.14 vs 47.05; MRPC 80.64 vs 80.15; QNLI 85.54 vs 85.41 | No clear effect |
| DPP-MAP inference instead of TopK inference with learned retriever (ablation) | EPR and CEIL retrievers · SST-5, MRPC, MNLI, CMSQA, MTOP, SMCalFlow | Accuracy/EM: DPP-EPR vs TopK-EPR: SST5 +2.72, MRPC +4.41, MNLI -0.97, CMSQA -1.23; DPP-CEIL vs TopK-CEIL: MRPC -1.22, CMSQA +0.08 | Mixed |
| Learning-free DPP-MAP inference vs TopK on BERT embeddings (ablation) | TopK-BERT vs DPP-BERT · SST-5, MRPC, MNLI, CMSQA, MTOP, SMCalFlow | Accuracy/EM: SST5 -0.46, MRPC +0.25, MNLI -2.55, CMSQA +1.88, MTOP +1.01, SMCalFlow -1.37 | Mixed |
| Increase candidate pool n from 50 to 800 at inference (KNN prefilter) (ablation) | CEIL inference · SST-5, MRPC, QNLI, GeoQuery, NL2Bash, MTOP | Avg. score: avg 67.96 (n=50) to 68.86 (n=100) to 69.32 (n=800); latency 30s to 118s | Mixed |
| Use random candidate sampling for training data (one-stage) vs two-stage top100+random (ablation) | CEIL training data construction · SST5 and MTOP | Accuracy/EM: SST5 35.97 (Rand C50) vs 47.05; MTOP 56.60 vs 67.43 | Hurt |
| Increase candidate subsets per instance from C10 to C50 in training data (ablation) | CEIL training data construction · SST5, GeoQuery, MTOP | Accuracy/EM: SST5 46.14 to 47.05; GeoQuery 67.86 to 73.21; MTOP 62.37 to 67.43 | Helped |
| CEIL retriever trained on GPT-Neo and transferred to Codex (learning-free there) | Codex 175B inference · SMCalFlow-CS 32-C | EM: +7.22 over TopK-BERT (55.28 vs 49.72) | Helped |
| CEIL retriever vs EPR on compositional GeoQuery splits | GPT-Neo 2.7B inferencer · GeoQuery Standard and Template splits | EM: Standard +4.64; Template +1.82; TMCD +0.00 | Helped |
| CEIL vs learning-free baselines on NLI tasks | GPT-Neo 2.7B · QNLI and MNLI | Accuracy: QNLI 85.41 vs Random 56.67; MNLI 71.74 vs TopK-BM25 40.68 | Helped |
Closest papers by meaning
- IDEAL: Influence-Driven Selective Annotations Empower In-Context Learners in Large Language Models (ICLR 2024 · similarity 0.88)
- In-Context Language Learning: Architectures and Algorithms (ICML 2024 · similarity 0.87)
- Sample Efficient Demonstration Selection for In-Context Learning (ICML 2025 · similarity 0.87)
- Understanding In-Context Learning from Repetitions (ICLR 2024 · similarity 0.87)
- GistScore: Learning Better Representations for In-Context Example Selection with Gist Bottlenecks (ICML 2024 · similarity 0.87)
- Towards Auto-Regressive Next-Token Prediction: In-context Learning Emerges from Generalization (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.