Research map / Large language models / Reasoning and chain of thought
Demonstration Selection for In-Context Learning via Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RDES, an RL-based (Q-learning and PPO variant) demonstration selection method that balances relevance and label-distribution diversity for in-context learning on intent classification and reasoning tasks.
Problem
Selecting demonstrations for in-context learning that are both relevant and diverse, since similarity-based selection can overfit and fixed strategies do not adapt to the task.
Approach
Frames demonstration selection as a finite-horizon MDP solved with tabular Q-learning or PPO, with a reward combining prediction correctness and a label-distribution diversity gain, while the LLM stays frozen.
Key results
- On closed-source models, RDES/C averages 0.838 on BANKING77 and 0.902 on CLINC150 versus 0.565 for the best baseline ADA on BANKING77 (table averages).
- On open-source models, RDES/C averages 0.845 on BANKING77 and 0.786 on LIU54, vs. ADA at 0.752 and 0.779 respectively.
- On supplementary reasoning tasks, RDES/C reaches 0.73 on GSM-8K with DeepSeek-R1-32B, while ADA reaches 0.36; with Qwen-2.5-72B, ADA attains the highest accuracy on several reasoning tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CoT reasoning to RDES (ablation) | RDES on closed-source LLMs · BANKING77 | accuracy: 0.737 -> 0.838 | Helped |
| Add CoT reasoning to RDES (ablation) | RDES on closed-source LLMs · CLINC150 | accuracy: 0.835 -> 0.902 | Helped |
| Add CoT reasoning to RDES (ablation) | RDES on closed-source LLMs · HWU64 | accuracy: 0.748 -> 0.872 | Helped |
| Add CoT reasoning to RDES (ablation) | RDES on closed-source LLMs · LIU54 | accuracy: 0.707 -> 0.824 | Helped |
| Add CoT reasoning to RDES (ablation) | RDES on open-source LLMs · BANKING77 | accuracy: 0.708 -> 0.845 | Helped |
| Add CoT reasoning to RDES (ablation) | RDES on open-source LLMs · CLINC150 | accuracy: 0.800 -> 0.731 | Hurt |
| Add CoT reasoning to RDES (ablation) | RDES on open-source LLMs · HWU64 | accuracy: 0.776 -> 0.853 | Helped |
| Add CoT reasoning to RDES (ablation) | RDES on open-source LLMs · LIU54 | accuracy: 0.686 -> 0.786 | Helped |
| Add the diversity mechanism (ablation) | RDES/C on closed-source LLMs · BANKING77 | accuracy: 0.600 -> 0.838 | Helped |
| Add the diversity mechanism (ablation) | RDES/C on closed-source LLMs · CLINC150 | accuracy: 0.770 -> 0.902 | Helped |
| Add the diversity mechanism (ablation) | RDES/C on closed-source LLMs · HWU64 | accuracy: 0.690 -> 0.872 | Helped |
| Add the diversity mechanism (ablation) | RDES/C on closed-source LLMs · LIU54 | accuracy: 0.640 -> 0.824 | Helped |
Surprising findings
- With Qwen-2.5-72B on GSM-8K and SST5, ADA outperforms RDES variants, and on several reasoning tables RDES/B and RDES/C underperform baselines like AES and ADA.
- Reported accuracy results vary inconsistently across k values, with several tables showing identical numbers for k=3 and k=7 and for k=5 and k=10.
Limitations
- Experiments limited to classification and a few reasoning tasks; generation and QA not evaluated.
- No user study; computational cost of many LLM calls acknowledged as a limitation.
- Method description is vague on state representation, action semantics, and how the LLM's prediction feeds back into the RL update; reported ablations show inconsistent diversity gains.
Experimental setup
Datasets: BANKING77 CLINC150 HWU64 LIU54 BigBenchHard (boolean expressions, web of lies) GSM-8K SST5
Benchmarks: BANKING77 CLINC150 HWU64 LIU54 BigBenchHard GSM-8K SST5
Metrics: accuracy
Baselines: Zero-Shot Prompting Knowledge Prompting Least-to-Most Prompting Chain of Thought Self-Refine Few-Shot Prompting Few-Shot with CoT Active Demonstration Selection (AES) Representative Demonstration Selection (RDS) ADAICL
Builds on: Proximal Policy Optimization Q-learning chain-of-thought prompting Adaptive Demonstration Selection (ADAICL) Representative demonstration selection (RDS)
Closest papers by meaning
- Comparative Analysis of Demonstration Selection Algorithms for In-Context Learning in Large Language Models (Student Abstract) (AAAI 2025 · similarity 0.89)
- Provoking Multi-modal Few-Shot LVLM via Exploration-Exploitation In-Context Learning (CVPR 2025 · similarity 0.88)
- Large Language Models are Demonstration Pre-Selectors for Themselves (ICML 2025 · similarity 0.88)
- Mixture of Demonstrations for In-Context Learning (NeurIPS 2024 · similarity 0.87)
- Dynamics-Predictive Sampling for Active RL Finetuning of Large Reasoning Models (ICLR 2026 · similarity 0.86)
- What Makes In-context Learning Effective for Mathematical Reasoning (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.