Research map / Large language models / Reasoning and chain of thought

Boosting of Thoughts: Trial-and-Error Problem Solving with Large Language Models

ICLR 2024 · Large language models › Reasoning and chain of thought · approach: Search and self-correction in reasoning · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Boosting of Thoughts (BoT), an automated, manual-annotation-free prompting framework that iteratively builds heterogeneous weighted binary thought trees, aggregates them into one chain, and accumulates LLM-generated error analysis and advice as experience in the prompt.

Problem

Complex multi-step reasoning with LLMs usually relies on hand-annotated chain-of-thought exemplars or task-specific tree prompts, which limits scalability; ineffective reasoning steps are typically discarded rather than learned from.

Approach

Start from a simple prompt with no examples, and iteratively enhance it with "experience" (error reports and revision advice produced by the same LLM analyzing an aggregated thought chain), using ensembles of shallow weighted binary trees with varied growth strategies and sampling settings, plus best-first or greedy aggregation.

Key results

• With GPT-4 and no human annotations, BoT reaches 97.1% on GSM8K (vs. SOTA 97) and 81.4% on AQuA (vs. 79.9); BoT+CoT averages 86.2 across four datasets vs. SOTA 88.7.
• On Game of 24 with GPT-4, BoT (no annotations) is 9.7 points above ToT.
• BoT is substantially weaker on MATH (62.5 for BoT, 66.3 for BoT+CoT vs. 84.3 SOTA), and gains on Llama-2 are limited (average +4.4% vs. +11.6% for GPT-4).

Measured effects reported in the paper

Surprising findings

• Including only issues without advice performs far worse than including advice, and accumulating too many experiences (no aggregation) hurts performance.
• Spurious LLM feedback (incorrectly claiming a wrong chain is correct) can be corrected in later iterations, so it does not get amplified.

Limitations

• Performance depends heavily on the LLM's ability to analyze reasoning chains; weaker models (Llama-2) gain little and MATH results lag well behind SOTA.
• Results are reported on a 100-game subset for Game of 24 and mostly on math tasks; only the ToT base structure was used (GoT and CR not evaluated).
• Relies on LLM self-evaluation, which may produce incorrect feedback.

Experimental setup

Datasets: MMLU SVAMP GSM8K AQuA MATH Game of 24

Benchmarks: SVAMP GSM8K AQuA MATH Game of 24

Metrics: solve rate (%)

Baselines: Standard IO CoT CoT-SC Complex CoT Zero-shot CoT Tree of Thoughts Progressive-Hint Prompting CSV

Builds on: Chain-of-thought prompting Tree of Thoughts Self-Refine Progressive-Hint Prompting Zero-shot CoT Self-Consistency

Closest papers by meaning

• THOUGHT PROPAGATION: AN ANALOGICAL APPROACH TO COMPLEX REASONING WITH LARGE LANGUAGE MODELS (ICLR 2024 · similarity 0.91)
• Large Language Models as Analogical Reasoners (ICLR 2024 · similarity 0.90)
• Enhancing Chain of Thought Prompting in Large Language Models via Reasoning Patterns (AAAI 2025 · similarity 0.90)
• Toward Adaptive Reasoning in Large Language Models with Thought Rollback (ICML 2024 · similarity 0.90)
• Chain of Thoughtlessness? An Analysis of CoT in Planning (NeurIPS 2024 · similarity 0.90)
• StrategyLLM: Large Language Models as Strategy Generators, Executors, Optimizers, and Evaluators for Problem Solving (NeurIPS 2024 · similarity 0.89)