Research map / Large language models / Agents, code and math

Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models

ICLR 2024 · Large language models › Agents, code and math · approach: Inductive and abstract reasoning evaluation · 22 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Step-Back Prompting, a two-step abstraction-then-reasoning prompting method in which LLMs first derive high-level concepts or principles and then reason over them, optionally with retrieval augmentation.

Problem

Complex multi-step reasoning in LLMs fails when tasks contain many low-level details, making intermediate reasoning steps error-prone.

Approach

Prompt the LLM to first ask a generic step-back question about high-level concepts or first principles (via few-shot demonstrations), retrieve or recite facts about them, and then answer the original question grounded on those abstractions.

Key results

• On MMLU Physics and Chemistry, PaLM-2L + Step-Back reaches 73.2% and 81.8%, versus 66.4% and 70.9% for the PaLM-2L baseline.
• On TimeQA, PaLM-2L + Step-Back + RAG reaches 68.7% versus 41.5% for PaLM-2L and 57.4% for PaLM-2L + RAG.
• On MuSiQue, PaLM-2L + Step-Back + RAG reaches 42.8% versus 35.5% for PaLM-2L; on StrategyQA it reaches 86.4%.

Measured effects reported in the paper

Surprising findings

• Chain-of-thought and take-a-deep-breath prompting give little or no gain on several STEM and knowledge QA tasks.
• Few-shot 1-shot baselines often perform worse than zero-shot baselines, e.g. on MuSiQue and StrategyQA.

Limitations

• Errors are dominated by reasoning and math errors rather than abstraction failures, so the reasoning step remains the bottleneck.
• Abstraction is not necessary or possible for all questions, such as simple factual lookups.
• Evaluation relies on PaLM-2L as an automatic judge for answer equivalence, with reported agreement with human ratings of 97-99%.

Experimental setup

Datasets: MMLU high-school physics MMLU high-school chemistry GSM8K TimeQA SituatedQA MuSiQue StrategyQA

Benchmarks: MMLU high-school physics MMLU high-school chemistry GSM8K TimeQA SituatedQA MuSiQue StrategyQA

Metrics: accuracy

Baselines: PaLM-2L PaLM-2L + CoT PaLM-2L + TDB PaLM-2L + RAG GPT-4 Llama2-70B Decomposed Prompting

Builds on: chain-of-thought prompting retrieval-augmented generation take a deep breath prompting recitation-augmented language models

Closest papers by meaning

• Large Language Models as Analogical Reasoners (ICLR 2024 · similarity 0.89)
• Boosting of Thoughts: Trial-and-Error Problem Solving with Large Language Models (ICLR 2024 · similarity 0.89)
• SCOUT: Teaching Pre-trained Language Models to Enhance Reasoning via Flow Chain-of-Thought (NeurIPS 2025 · similarity 0.89)
• Enhancing Chain of Thought Prompting in Large Language Models via Reasoning Patterns (AAAI 2025 · similarity 0.89)
• DOTS: Learning to Reason Dynamically in LLMs via Optimal Reasoning Trajectories Search (ICLR 2025 · similarity 0.88)
• On the self-verification limitations of large language models on reasoning and planning tasks (ICLR 2025 · similarity 0.88)