Research map / Large language models / Reasoning and chain of thought
Large Language Models as Analogical Reasoners
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes analogical prompting, where an LLM self-generates relevant exemplars and high-level knowledge in a single prompt before solving the target problem, requiring no labeled exemplars or retrieval.
Problem
Chain-of-thought prompting either uses generic reasoning instructions (0-shot CoT), which may be insufficient for complex tasks, or requires costly labeled exemplars (few-shot CoT) and retrieval machinery for relevant ones.
Approach
Prompt the LLM to recall three relevant, distinct problems with solutions (and optionally a tutorial on core concepts) in the same prompt before solving the target problem, so exemplars are self-generated and tailored per problem, inspired by human analogical reasoning.
Key results
- On MATH with GPT-3.5-turbo, analogical prompting reaches 37.3% vs 34.9% for 5-shot CoT and 33.9% for 0-shot CoT.
- On Codeforces (50 problems), self-generated knowledge + exemplars gives GPT-4 Acc@1 19% vs 16% for 0-shot and 17% for 3-shot CoT.
- Combining with self-consistency raises GSM8K from 77.8% to 85.3% and MATH from 37.3% to 46.0%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace fixed few-shot CoT exemplars with self-generated relevant exemplars | GPT3.5-turbo · GSM8K | accuracy: 76.7% -> 77.8% | Helped |
| Replace fixed few-shot CoT exemplars with self-generated relevant exemplars | GPT3.5-turbo · MATH | accuracy: 34.9% -> 37.3% | Helped |
| Add self-generated knowledge to self-generated exemplars | GPT3.5-turbo-16k · Codeforces | Acc@1: 13% -> 15% | Helped |
| Add self-generated knowledge to self-generated exemplars | GPT3.5-turbo-16k · Codeforces | Acc@10: 25% -> 29% | Helped |
| Replace fixed few-shot CoT exemplars with self-generated exemplars | GPT3.5-turbo · Temporal sequences | accuracy: 58.0% -> 57.6% | Hurt |
| Encourage diversity among self-generated exemplars (ablation) | Analogical prompting · GSM8K | accuracy: 75.9% -> 77.8% | Helped |
| Encourage diversity among self-generated exemplars (ablation) | Analogical prompting · MATH | accuracy: 35.2% -> 37.3% | Helped |
| Generate knowledge before rather than after exemplars (ablation) | Analogical prompting · Codeforces | Acc@1: 14% -> 15% | Helped |
| Generate knowledge before rather than after exemplars (ablation) | Analogical prompting · Codeforces | Acc@10: 27% -> 29% | Helped |
| Add self-consistency to analogical prompting (ablation) | GPT3.5-turbo · GSM8K | accuracy: 77.8% -> 85.3% | Helped |
| Add self-consistency to analogical prompting (ablation) | GPT3.5-turbo · MATH | accuracy: 37.3% -> 46.0% | Helped |
| Vary the number of self-generated exemplars from 1 to 5 (ablation) | GPT3.5-turbo analogical prompting · GSM8K | accuracy: 76.1; 77.0; 77.5; 77.3; 77.8 | Mixed |
Surprising findings
- Self-generated exemplars underperform retrieved exemplars with small models (text-curie-001, text-davinci-001) and only win at text-davinci-002 and larger.
- Generating knowledge gives little gain on GSM8K and BIG-Bench, where tasks are judged less complex.
Limitations
- Increased inference computation due to longer generated outputs.
- Self-generation fails when the LLM is weak or lacks relevant knowledge, so it suits only stronger models.
- Generated exemplars may not generalize to harder new problems, and results are sensitive to prompt phrasing.
Experimental setup
Datasets: GSM8K MATH BIG-Bench Codeforces
Benchmarks: GSM8K MATH Codeforces BIG-Bench word sorting BIG-Bench logical deduction five objects BIG-Bench temporal sequences BIG-Bench reasoning about colored objects BIG-Bench formal fallacies
Metrics: accuracy Acc@1 Acc@10
Baselines: 0-shot 0-shot CoT few-shot CoT few-shot retrieved CoT self-consistency
Builds on: chain-of-thought prompting few-shot CoT zero-shot CoT Sentence-BERT retrieval recitation-augmented generation
Closest papers by meaning
- Enhancing Chain of Thought Prompting in Large Language Models via Reasoning Patterns (AAAI 2025 · similarity 0.93)
- Scaling Code-Assisted Chain-of-Thoughts and Instructions for Model Reasoning (NeurIPS 2025 · similarity 0.90)
- When Thinking Fails: The Pitfalls of Reasoning for Instruction-Following in LLMs (NeurIPS 2025 · similarity 0.90)
- Chain-of-Thought Reasoning Without Prompting (NeurIPS 2024 · similarity 0.90)
- Boosting of Thoughts: Trial-and-Error Problem Solving with Large Language Models (ICLR 2024 · similarity 0.90)
- THOUGHT PROPAGATION: AN ANALOGICAL APPROACH TO COMPLEX REASONING WITH LARGE LANGUAGE MODELS (ICLR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.