Research map / Large language models / Language, safety and interpretability

In-Context Principle Learning from Mistakes

ICML 2024 · Large language models › Language, safety and interpretability · approach: In-context learning analysis · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LEAP, which has the LLM make mistakes zero-shot on a few labeled examples, writes low- and high-level natural-language principles from those mistakes, and prepends them to the standard few-shot prompt at test time.

Problem

Standard in-context learning only learns from correct input-output examples, ignoring the information in the model's mistakes on those same few examples.

Approach

Intentionally induce mistakes on the few-shot examples via sampled zero-shot CoT, have the same LLM articulate explicit task-level principles by contrasting mistaken reasoning with the correct answer, then condition test-time few-shot prompts on these principles; the principle learning is done once per task.

Key results

• LEAP improves GPT-4 on DROP by 7.5 points over few-shot CoT (72.05 to 79.55).
• LEAP improves GPT-4 on HotpotQA by 3.3 points (36.35 to 39.65 with high-level principles).
• In 37 of 42 task-model combinations on BIG-Bench Hard, at least one LEAP variant beats few-shot CoT.

Measured effects reported in the paper

Surprising findings

• Open-source Llama-2-chat-70B did not benefit from principles, even when they were generated by GPT-4.
• Contrastive CoT's reported gains on GSM8K did not reproduce after fixing answer-extraction code, performing similar to or worse than few-shot CoT.

Limitations

• Open-source models did not follow the learned principles at test time, so gains appear limited to strong proprietary models.
• Gemini Pro on HotpotQA underperformed the baseline because its principles were overly example-specific and verbose.
• Gains on math benchmarks are modest and inconsistent across models (e.g., Claude-2 GSM8K and Gemini Pro MATH).

Experimental setup

Datasets: HotpotQA DROP GSM8K MATH BIG-Bench Hard

Benchmarks: HotpotQA DROP GSM8K MATH BIG-Bench Hard

Metrics: accuracy

Baselines: Few-shot CoT Zero-shot CoT Contrastive CoT OPRO Self-Refine MemPrompt EvoPrompt

Builds on: chain-of-thought prompting Self-Refine OPRO Contrastive Chain-of-Thought

Closest papers by meaning

• The Surprising Effectiveness of Test-Time Training for Few-Shot Learning (ICML 2025 · similarity 0.89)
• Evolving Prompts In-Context: An Open-ended, Self-replicating Perspective (ICML 2025 · similarity 0.88)
• StrategyLLM: Large Language Models as Strategy Generators, Executors, Optimizers, and Evaluators for Problem Solving (NeurIPS 2024 · similarity 0.87)
• Is In-Context Learning Sufficient for Instruction Following in LLMs? (ICLR 2025 · similarity 0.87)
• Many-Shot CoT-ICL: Making In-Context Learning Truly Learn (ICML 2026 · similarity 0.87)
• Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.87)