Research map / Large language models / Language, safety and interpretability
What Makes In-context Learning Effective for Mathematical Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretically bounds how in-context demonstrations affect LLM math reasoning loss, then proposes LMS3, a demonstration selection method balancing LLM-oriented semantic similarity and inference stability with a rejection mechanism that falls back to zero-shot.
Problem
Few-shot demonstrations can hurt LLM mathematical reasoning and their effect is unreliable; existing work lacks precise theory on when demonstrations help.
Approach
Derives via a linearized attention/influence-function analysis that demonstrations reduce test loss when they are semantically close (measured in the inference LLM's representation space) and the LLM is stable on them; selects demonstrations by the product of similarity and stability, and rejects demonstrations entirely when similarity ranks poorly.
Key results
- On one-shot Llama3-8B, LMS3 reaches 0.837 on MATH vs 0.324 zero-shot and 0.817 for best baseline (Similar-ICL variants ~0.80-0.82 range), and is the only method consistently above zero-shot across MAWPS, GSM8K, and MATH.
- LMS3 achieves 0.444 (Llama2-13B) and 0.650 (Llama3-8B) on CommonsenseQA, outperforming all baselines.
- Adding the rejection mechanism to baselines improves them on most datasets (e.g., IDS on GSM8K with Llama3-8B rises from 0.808 to 0.823).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LMS3-selected one-shot demonstrations instead of zero-shot prompting | Llama2-13B · MAWPS | answer accuracy: 0.835± 0.009 -> 0.854∗± 0.008 | Helped |
| Use LMS3-selected one-shot demonstrations instead of zero-shot prompting | Llama2-13B · GSM8K | answer accuracy: 0.414± 0.004 -> 0.447∗± 0.014 | Helped |
| Use LMS3-selected one-shot demonstrations instead of zero-shot prompting | Llama2-13B · MATH | answer accuracy: 0.096± 0.005 -> 0.124∗± 0.003 | Helped |
| Use LMS3-selected one-shot demonstrations instead of zero-shot prompting | Llama3-8B · MAWPS | answer accuracy: 0.951± 0.004 -> 0.966± 0.010 | Helped |
| Use LMS3-selected one-shot demonstrations instead of zero-shot prompting | Llama3-8B · GSM8K | answer accuracy: 0.820± 0.016 -> 0.837∗± 0.011 | Helped |
| Use LMS3-selected one-shot demonstrations instead of zero-shot prompting | Llama3-8B · MATH | answer accuracy: 0.324± 0.022 -> 0.353∗± 0.002 | Helped |
| Apply LMS3-selected demonstrations to ChatGPT | ChatGPT · GSM8K | answer accuracy: 0.800 -> 0.862 | Helped |
| Apply LMS3-selected demonstrations to GPT-4 | GPT-4 · MATH | answer accuracy: 0.758 -> 0.752 | Hurt |
| Add the demonstration rejection mechanism to LMS3 rather than retaining demonstrations at λ = 100% (ablation) | LMS3 · one-shot reasoning on MAWPS, GSM8K, and MATH | answer accuracy | Helped |
| Add the rejection mechanism to baseline demonstration selectors (ablation) | Llama3-8B demonstration selection · MAWPS, GSM8K, and MATH | answer accuracy: 0.920 -> 0.932 | Helped |
Surprising findings
- Few-shot demonstrations can reduce accuracy below zero-shot; e.g., ChatGPT on MATH declines with one-shot in Figure 1.
- With the rejection mechanism disabled (lambda=100%), LMS3 falls below zero-shot accuracy.
Limitations
- Theory relies on linearized attention approximation and Lipschitz/positive-definite Hessian assumptions; the rejection threshold is a heuristic relative rank rather than the derived absolute condition.
- Gains on GSM8K and MAWPS are small, and the optimal lambda differs across LLMs requiring tuning.
- Evaluated only on two open Llama backbones for selection; transfer to GPT models is with demonstrations selected by Llama3-8B.
Experimental setup
Datasets: MAWPS GSM8K MATH CommonsenseQA
Benchmarks: MAWPS GSM8K MATH CommonsenseQA
Metrics: answer accuracy
Baselines: Random Best-validate TF-IDF BM25 T5 BGEM3 OpenAI embedding SPELL Influence IDS
Builds on: influence functions (Koh & Liang) linear attention as meta-gradient (Dai et al.) InfICL SPELL IDS
Closest papers by meaning
- Are Human-generated Demonstrations Necessary for In-context Learning? (ICLR 2024 · similarity 0.90)
- Comparative Analysis of Demonstration Selection Algorithms for In-Context Learning in Large Language Models (Student Abstract) (AAAI 2025 · similarity 0.90)
- Many-Shot CoT-ICL: Making In-Context Learning Truly Learn (ICML 2026 · similarity 0.90)
- Large Language Models are Demonstration Pre-Selectors for Themselves (ICML 2025 · similarity 0.89)
- Putnam-AXIOM: A Functional & Static Benchmark for Measuring Higher Level Mathematical Reasoning in LLMs (ICML 2025 · similarity 0.89)
- The Surprising Effectiveness of Test-Time Training for Few-Shot Learning (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.