Research map / Large language models / Language, safety and interpretability
Comparative Analysis of Demonstration Selection Algorithms for In-Context Learning in Large Language Models (Student Abstract)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Empirically benchmarks six demonstration selection algorithms for in-context learning with LLaMa3-8B across five GLUE/CMSQA/SWAG datasets, measuring both accuracy and selection/inference time.
Problem
The effectiveness and computational efficiency of demonstration selection algorithms for in-context learning are poorly understood, making it hard to choose or improve them.
Approach
A systematic side-by-side comparison of existing demonstration selection methods (CBDS, RD-direct, RD-channel, LLM Retriever, UPRISE, OpenICL TopK) against random selection, reporting accuracy for k in {4, 8, 10, 20} and per-sample selection and inference time, with offline costs excluded.
Key results
- CBDS or UPRISE achieved the best accuracy in 14 of 20 dataset-k combinations, but CBDS selection took about 5.4 seconds per sample and UPRISE about 3 seconds.
- Some sophisticated selectors (e.g., RD-direct, RD-channel) do not consistently beat OpenICL random selection; for example, on SST2 at k=4 RD-direct scores 56.4 vs. random 86.2.
- Increasing the number of demonstrations k does not monotonically improve accuracy; the trend varies by task and algorithm.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a sophisticated demonstration selection algorithm instead of random selection | Demonstration selection for LLaMA3-8B · Across the five evaluated datasets | accuracy | Mixed |
| Use CBDS or UPRISE rather than the other compared algorithms | Demonstration selection for LLaMA3-8B · 20 combinations of k and datasets | accuracy: 14 cases | Helped |
| Use CBDS instead of Random Selection | Demonstration selection for LLaMA3-8B · GLUE-MRPC | selection time (seconds): 5.24 ∗10−6 -> 5.44 | Hurt |
| Use UPRISE instead of Random Selection | Demonstration selection for LLaMA3-8B · GLUE-MRPC | selection time (seconds): 5.24 ∗10−6 -> 3.06 | Hurt |
| Present demonstrations using the channel approach instead of the direct approach | RD · Across evaluated datasets | accuracy | Mixed |
| Increase the number of demonstrations | Demonstration selection algorithms · Across tasks and algorithms | accuracy | Mixed |
Surprising findings
- Some sophisticated demonstration selection algorithms fail to outperform random selection in certain scenarios.
- Increasing the number of demonstrations does not consistently improve performance.
Limitations
- Evaluation limited to a single LLM (LLaMa3-8B) and five datasets.
- Only accuracy and wall-clock time are reported; no statistical significance or variance analysis is evident.
Experimental setup
Datasets: GLUE-MRPC GLUE-QNLI GLUE-SST2 CMSQA SWAG
Benchmarks: GLUE-MRPC GLUE-QNLI GLUE-SST2 CMSQA SWAG
Metrics: accuracy selection time (seconds) inference time (seconds)
Baselines: CBDS RD-direct RD-channel LLM Retriever UPRISE OpenICL TopK OpenICL Random
Builds on: Rethinking the role of demonstrations (RD) UPRISE OpenICL
Closest papers by meaning
- Large Language Models are Demonstration Pre-Selectors for Themselves (ICML 2025 · similarity 0.92)
- What Makes In-context Learning Effective for Mathematical Reasoning (ICML 2025 · similarity 0.90)
- Mixture of Demonstrations for In-Context Learning (NeurIPS 2024 · similarity 0.90)
- Demonstration Selection for In-Context Learning via Reinforcement Learning (ICML 2025 · similarity 0.89)
- Sample Efficient Demonstration Selection for In-Context Learning (ICML 2025 · similarity 0.88)
- MEND: Meta Demonstration Distillation for Efficient and Effective In-Context Learning (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.