Research map / Large language models / Language, safety and interpretability

Many-Shot In-Context Learning

NeurIPS 2024 · Large language models › Language, safety and interpretability · approach: In-context learning analysis

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Studies many-shot in-context learning (up to ~8192 shots, up to 1M tokens) with Gemini 1.5 Pro across translation, summarization, planning, reward modeling, reasoning and classification, and introduces reinforced and unsupervised ICL to reduce reliance on human-written rationales.

Problem

Few-shot in-context learning is limited by short context windows, so it is unclear how LLM performance scales when prompted with hundreds to thousands of examples, and whether many-shot ICL can reduce dependence on human-written demonstrations.

Approach

Scales ICL to the many-shot regime with long-context LLMs and introduces Reinforced ICL (model-generated, correctness-filtered rationales as demonstrations) and Unsupervised ICL (only problems, no solutions) as alternatives to human-written shots.

Key results

• Many-shot ICL with Gemini 1.5 Pro improves English-to-Bemba chrF2++ by 15.3% and English-to-Kurdish by 4.5% over 1-shot at 997 shots, surpassing NLLB and Google Translate SOTA.
• Reinforced ICL on BIG-Bench Hard averages 83% versus 72.1% for human-written CoT prompting and 77.1% for Unsupervised ICL.
• On GPQA, 125-shot ICL with ground-truth rationales nearly matches Claude-3 Opus, and both ground-truth and model-generated 125-shot prompts surpass the 40.4% of Claude-3 Sonnet.

Measured effects reported in the paper

Surprising findings

• Performance on MATH degrades beyond about 125 shots for standard ICL, while negative log-likelihood continues to decrease, so NLL is not a reliable predictor of downstream performance.
• Many-shot ICL on sequential parity surpasses a GPT-2 Medium model trained from scratch on 20x more examples, despite parity being a task where nearest-neighbour reasoning should fail.

Limitations

• Main experiments mostly use only Gemini 1.5 Pro; generalization to other long-context LLMs is only preliminarily tested.
• Performance sometimes degrades with more shots (e.g., MATH, GPQA at 250 shots), and the cause is not understood.
• Reinforced ICL can admit false positives (correct answer with incorrect reasoning).

Experimental setup

Datasets: MATH GSM8K GPQA BIG-Bench Hard FLORES-200 XSum XLSum Financial PhraseBank PDDL Logistics generated problems

Benchmarks: MATH500 GSM8K GPQA diamond BIG-Bench Hard (8 tasks) FLORES-200 English-Bemba and English-Kurdish XSum XLSum Financial PhraseBank Logistics PDDL planning

Metrics: accuracy chrF2++ ROUGE-L success rate negative log-likelihood best-of-4 accuracy

Baselines: few-shot ICL with human-written solutions Gemini 1.5 Pro 1-shot GPT-4-Turbo Claude-3-Opus Gemini 1.5 Flash PEGASUS mT5 Fast-Downward GPT-2 Medium trained from scratch k-nearest neighbors full supervised fine-tuning

Builds on: Reinforced Self-Training (ReST) chain-of-thought prompting KV caching STaR-style self-generated rationales

Closest papers by meaning

• From Few to Many: Self-Improving Many-Shot Reasoners Through Iterative Optimization and Generation (ICLR 2025 · similarity 0.95)
• MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learning (ICML 2025 · similarity 0.92)
• MIR-Bench: Can Your LLM Recognize Complicated Patterns via Many-Shot In-Context Reasoning? (NeurIPS 2025 · similarity 0.91)
• Revisiting In-context Learning Inference Circuit in Large Language Models (ICLR 2025 · similarity 0.90)
• Many-Shot CoT-ICL: Making In-Context Learning Truly Learn (ICML 2026 · similarity 0.90)
• The Surprising Effectiveness of Test-Time Training for Few-Shot Learning (ICML 2025 · similarity 0.89)