Research map / Large language models / Language, safety and interpretability
Are Human-generated Demonstrations Necessary for In-context Learning?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes self-contemplation prompting (SEC), where an LLM first generates its own few-shot demonstrations (optionally with reasoning) for each test input and then answers, removing the need for human-crafted demonstrations.
Problem
In-context learning depends heavily on which human-crafted demonstrations are chosen, and crafting demonstrations (especially reasoning rationales) is laborious or infeasible for many tasks.
Approach
Instead of supplying hand-crafted demonstrations, the LLM is prompted to generate demonstrations tailored to each test input (with reasoning in the CoT variant) and then uses these self-generated demonstrations to produce the final answer, requiring two queries per sample.
Key results
- On GPT-3.5, CoT-SEC reaches 33.5% on MATH vs 31.2% for CoT-ICL and 71.4% on MMLU vs 69.6% for CoT-ICL.
- On GPT-3.5, vanilla SEC reaches 75.6% on HumanEval, above vanilla ICL at 73.8%, while zero-shot gets 48.8%.
- SEC underperforms ICL on a 3-digit base-5 addition test set, indicating reliance on in-distribution generation quality.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace human-crafted CoT demonstrations with LLM-generated demonstrations | GPT-3.5 · MATH Test | accuracy: 31.2 -> 33.5 | Helped |
| Replace human-crafted vanilla demonstrations with LLM-generated demonstrations | GPT-3.5 · HumanEval Test | accuracy: 73.8 -> 75.6 | Helped |
| Replace human-crafted CoT demonstrations with LLM-generated demonstrations | GPT-3.5 · ARC Test | accuracy: 87.9 -> 86.9 | Hurt |
| Replace human-crafted CoT demonstrations with LLM-generated demonstrations | GPT-3.5 · GSM8K Test | accuracy: 77.4 -> 77.0 | Hurt |
| Use CoT-SEC instead of CoT-ICL | GPT-4 · ARC Test | accuracy: 95.1 -> 96.2 | Helped |
| Use CoT-SEC instead of CoT-ICL | GPT-4 · MATH Test | accuracy: 42.3 -> 41.9 | Hurt |
| Use CoT-SEC instead of CoT-ICL | GPT-4 · GSM8K Test | accuracy: 92.0 -> 92.1 | Helped |
| Replace human-crafted demonstrations with model-generated demonstrations | SEC vs. ICL · MATH Test | accuracy: 31.2 -> 33.5; 42.3 -> 41.9; 7.4 -> 7.5 | Mixed |
| Increase the number of generated demonstrations (ablation) | SEC · HumanEval Test | accuracy | Hurt |
| Use SEC with a model that is not strong enough instead of ICL (ablation) | smaller GPT-3.5-family models · GSM8K | accuracy | Hurt |
| Use SEC rather than ICL on rare 3-digit base-5 addition problems | SEC · 3-digit base-5 addition problems | accuracy: 28.0 -> 27.0; 27.0 -> 24.0 | Hurt |
| Use CoT-SEC rather than zero-shot prompting | GPT-3.5 · GSM8K Test | accuracy: 73.4 -> 77.0 | Helped |
Surprising findings
- Approximately 22% of GSM8K samples have opposite correctness between CoT-SEC and CoT-ICL, while incorrect self-generated demonstrations often still yield correct final answers.
- On HumanEval, SEC performance slightly decreases as the number of shots increases, and the self-generated demonstrations are simpler than canonical solutions.
Limitations
- Performance degrades on weaker models, where SEC underperforms ICL.
- Self-generated demonstrations can be low quality or mismatched to the test question, and rounding answers to integers for GSM8K may mislead the model.
- Underperforms ICL on an out-of-distribution base-5 addition test set.
Experimental setup
Datasets: GSM8K MATH ARC MMLU C-Eval HumanEval
Benchmarks: MATH GSM8K ARC MMLU C-Eval HumanEval
Metrics: exact match accuracy pass@1 accuracy
Baselines: Zero-shot Zero-shot CoT Vanilla ICL CoT-ICL Auto-CoT
Builds on: chain-of-thought prompting zero-shot chain-of-thought Auto-CoT kNN prompting
Closest papers by meaning
- What Makes In-context Learning Effective for Mathematical Reasoning (ICML 2025 · similarity 0.90)
- MEND: Meta Demonstration Distillation for Efficient and Effective In-Context Learning (ICLR 2024 · similarity 0.90)
- DETAIL: Task DEmonsTration Attribution for Interpretable In-context Learning (NeurIPS 2024 · similarity 0.89)
- Reward Is Enough: LLMs Are In-Context Reinforcement Learners (ICLR 2026 · similarity 0.89)
- Mixture of Demonstrations for In-Context Learning (NeurIPS 2024 · similarity 0.88)
- Many-Shot CoT-ICL: Making In-Context Learning Truly Learn (ICML 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.