Research map / Large language models / Language, safety and interpretability
Easier to Judge than to Find: Predicting In-Context Learning Success for Demonstration Selection
Haochun Wang, Chaofen Yang, Jiatong Liu, Jingbo Wang, Zewen Qiang, Sendong Zhao, Bing Qin, Ting Liu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DiSP judge-based selection instead of zero-shot | Llama3-8B · 5-dataset average | accuracy: 66.4 -> 77.6 | Improved |
| use DiSP judge-based selection instead of zero-shot | Qwen2.5-7B · 5-dataset average | accuracy: 69.8 -> 82.9 | Improved |
| use DiSP instead of strongest baseline | BM25 · Llama3-8B average | accuracy: 74.8 -> 77.6 | Improved |
| use DiSP instead of strongest baseline | BM25 · Qwen2.5-7B average | accuracy: 80.5 -> 82.9 | Improved |
| use DiSP instead of Se2 on TREC | Se2 · TREC, Llama3-8B | accuracy: 68.2 -> 79.2 | Improved |
| use learned selectors Se2 and SeDPO instead of zero-shot | zero-shot · TREC, Llama3-8B | accuracy: 70.5 -> 67.5 | Hurt |
| add demonstrations to SST-2 queries | zero-shot · SST-2, Llama3-8B | accuracy: 94.7 -> 95.4 | No clear effect |
| use DiSP over Uprise on training cost | Uprise · training time | minutes: 13.4 -> 6.7 | Improved |
| use DiSP over SeDPO on training cost | SeDPO · training time | minutes: 157.1 -> 6.7 | Improved |
| train one pooled judge across all datasets instead of per-dataset judges (ablation) | DiSP, l1 · Llama3-8B | accuracy: 91.5 -> 90.6 | Hurt |
| train one pooled judge across all datasets instead of per-dataset judges (ablation) | DiSP, l2 · Llama3-8B | accuracy: 65.7 -> 60.5 | Hurt |
| train one pooled judge across all datasets instead of per-dataset judges (ablation) | DiSP, l3 · Llama3-8B | accuracy: 55.8 -> 52.4 | Hurt |
Closest papers by meaning
- Comparative Analysis of Demonstration Selection Algorithms for In-Context Learning in Large Language Models (Student Abstract) (AAAI 2025 · similarity 0.90)
- Sample Efficient Demonstration Selection for In-Context Learning (ICML 2025 · similarity 0.89)
- Mixture of Demonstrations for In-Context Learning (NeurIPS 2024 · similarity 0.88)
- Large Language Models are Demonstration Pre-Selectors for Themselves (ICML 2025 · similarity 0.88)
- Demonstration Selection for In-Context Learning via Reinforcement Learning (ICML 2025 · similarity 0.87)
- Disentangling Latent Shifts of In-Context Learning with Weak Supervision (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
