Research map / Large language models / Reasoning and chain of thought
Instance-adaptive Zero-shot Chain-of-Thought Prompting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IAP, an instance-adaptive zero-shot CoT method that selects per-question prompts using information-flow saliency scores (question-to-prompt, question-to-rationale, prompt-to-rationale) to distinguish good from bad reasoning.
Problem
Task-level zero-shot CoT prompts are uniformly applied to all instances, but one prompt is not optimal for every instance; it is unclear why some zero-shot CoT reasoning succeeds and others fail.
Approach
Analyze zero-shot CoT via attention-gradient saliency to identify information-flow patterns (question-to-prompt and question-to-rationale flows) that separate correct from incorrect rationales, then use these saliency scores to choose a prompt per instance (IAP-ss sequential substitution, IAP-mv majority vote over top-k scores).
Key results
- IAP-mv raises LLaMA-3-8B-Instruct GSM8K accuracy from 64.52% (best single prompt #1) to 66.34%, and Qwen-14B-Chat from 60.50% to 62.81%.
- IAP-mv improves Qwen-14B-Chat on SVAMP to 73.33% vs 72.00% for the best single prompt, and on CSQA improves LLaMA-3 accuracy by about 3.4 points over the best task-level prompt.
- With LLaMA-3-70B-Instruct, IAP-mv reaches 89.84% on GSM8K versus 89.16% for the best single prompt (#2).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use IAP-mv instead of the strongest reported non-IAP method | LLaMA-3-8B-Instruct · GSM8K | accuracy: 65.96 -> 66.34 | Helped |
| Use IAP-mv instead of the strongest reported non-IAP method | LLaMA-3-8B-Instruct · SVAMP | accuracy: 77.00 -> 77.33 | Helped |
| Use IAP-mv instead of the strongest reported non-IAP method | LLaMA-3-8B-Instruct · Causal Judgement | accuracy: 18.18 -> 19.25 | Helped |
| Use IAP-mv instead of the strongest reported non-IAP method | LLaMA-3-8B-Instruct · Tracking Shuffled Objects | accuracy: 36.00 -> 42.40 | Helped |
| Use IAP-mv instead of the strongest reported non-IAP method | LLaMA-3-8B-Instruct · CSQA | accuracy: 65.03 -> 68.39 | Helped |
| Use IAP-mv instead of IAP-ss | LLaMA-3-8B-Instruct · MMLU | accuracy: 56.49 -> 59.65 | Helped |
| Use IAP-mv instead of the strongest task-level prompt | Qwen-14B-Chat · GSM8K | accuracy: 60.50 -> 62.81 | Helped |
| Use IAP-mv instead of the strongest task-level prompt | Qwen-14B-Chat · SVAMP | accuracy: 72.00 -> 73.33 | Helped |
| Use IAP-mv instead of the strongest task-level prompt | Qwen-14B-Chat · Causal Judgement | accuracy: 27.50 -> 29.95 | Helped |
| Use IAP-mv instead of the strongest task-level prompt | Qwen-14B-Chat · Tracking Shuffled Objects | accuracy: 23.20 -> 25.60 | Helped |
| Use IAP-mv instead of the strongest task-level prompt | Qwen-14B-Chat · CSQA | accuracy: 63.23 -> 65.68 | Helped |
| Use IAP-mv instead of the strongest task-level prompt | Qwen-14B-Chat · MMLU | accuracy: 76.84 -> 78.95 | Helped |
Surprising findings
- Answer majority vote over all nine prompts performs poorly on several tasks, while IAP-mv substantially outperforms it, indicating most prompts yield wrong answers for a given question.
- IAP gave no salient gains on LLaMA-2-13B-Chat, attributed to irregular output formats.
Limitations
- Analysis relies on answer-step detection via regular expressions, which fails on some irregular outputs.
- Findings may not generalize to all LLMs; the authors acknowledge the analysis cannot meet all models.
- IAP-ss thresholds are model- and dataset-specific and IAP-mv costs evaluation of all candidate prompts.
Experimental setup
Datasets: GSM8K SVAMP CommonsenseQA MMLU Causal Judgement (BBH) Tracking Shuffled Objects (BBH)
Benchmarks: GSM8K SVAMP Causal Judgement Tracking Shuffled Objects CommonsenseQA MMLU
Metrics: accuracy
Baselines: Answer majority vote (AMV) OPPR Self-Discover
Builds on: Zero-shot CoT OPRO Self-Discover saliency score analysis for in-context learning
Closest papers by meaning
- Enhancing Chain of Thought Prompting in Large Language Models via Reasoning Patterns (AAAI 2025 · similarity 0.93)
- Chain-of-Thought Reasoning Without Prompting (NeurIPS 2024 · similarity 0.91)
- SCOUT: Teaching Pre-trained Language Models to Enhance Reasoning via Flow Chain-of-Thought (NeurIPS 2025 · similarity 0.90)
- CoT Vectors: Transferring and Probing the Reasoning Mechanisms of LLMs (ICLR 2026 · similarity 0.90)
- Scaling Code-Assisted Chain-of-Thoughts and Instructions for Model Reasoning (NeurIPS 2025 · similarity 0.90)
- Large Language Models as Analogical Reasoners (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.