Research map / Large language models / Retrieval and knowledge
RATT: A Thought Structure for Coherent and Correct LLM Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RATT, a thought-tree prompting method that combines per-step lookahead/planning with RAG-based fact-checking and correction, evaluated with GPT-3.5 Turbo on HumanEval, creative writing, TruthfulQA hallucination detection, and Game of 24.
Problem
LLM reasoning with thought structures (CoT, ToT) lacks timely local fact-checking, and RAT lacks global planning/lookahead, so errors propagate and strategies are suboptimal.
Approach
At each node of a thought tree, generate multiple candidate thoughts, merge them into one node, retrieve documents from an external library using the prompt and node embeddings, and correct/enrich the node with retrieved facts, iterating so that factual correction and strategic lookahead happen online rather than after reasoning.
Key results
- On HumanEval, RATT reaches 69.70% pass@1 versus 59.27% for RAT and 50.49% for IO with GPT-3.5 Turbo.
- On Game of 24 (100 games), RATT achieves 31.8% success versus 14.0% for ToT and 9.0% for RAT.
- Creative writing scores are judged by GPT-4 for half of the reviews and human experts for the rest; the paper reports RATT outperforming baselines across dimensions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace RAT with RATT | GPT-3.5 Turbo · HumanEval | pass@1: 59.27% -> 69.70% | Helped |
| Replace RAT with RATT | GPT-3.5 Turbo · HumanEval | pass@5: 80.49% -> 83.53% | Helped |
| Replace ToT with RATT | GPT-3.5 Turbo · Game of 24 | success rate: 14.0 -> 31.8 | Helped |
| Increase generation steps from 3 to 5 (ablation) | RATT · Game of 24 | success rate: 25.9 -> 31.8 | Helped |
Surprising findings
- RAT performs worse than ToT and even IO on Game of 24 (9.0% vs. 14.0% and 7.6%), indicating retrieval-based correction alone does not help numerical search.
Limitations
- High computational demands from iterative multi-node generation and retrieval.
- Limited scalability with very complex tasks or large external libraries.
- Performance depends heavily on prompt quality and external library quality.
Experimental setup
Datasets: HumanEval codeparrot/github-jupyter English Wikipedia TruthfulQA 4nums Game of 24 puzzles
Benchmarks: HumanEval TruthfulQA Game of 24 Creative Writing (custom rubric)
Metrics: pass@1 pass@5 BLEU ROUGE GPT-judge truthfulness success rate soundness/relevance/coherence/clarity score
Baselines: IO CoT CoT-SC ToT RAT RAG 1-shot RAG 5-shot
Builds on: Tree of Thoughts Retrieval Augmented Thoughts Chain-of-Thought Retrieval-Augmented Generation
Closest papers by meaning
- Learning to Reason via Mixture-of-Thought for Logical Reasoning (ICLR 2026 · similarity 0.90)
- Forest-of-Thought: Scaling Test-Time Compute for Enhancing LLM Reasoning (ICML 2025 · similarity 0.90)
- Don’t Think Longer, Think Wisely: Optimizing Thinking Dynamics for Large Reasoning Models (NeurIPS 2025 · similarity 0.90)
- Accelerating Large Language Model Reasoning via Speculative Search (ICML 2025 · similarity 0.90)
- MindMap: Constructing Evidence Chains for Multi-Step Reasoning in Large Language Models (AAAI 2024 · similarity 0.89)
- Unveiling the Magic of Code Reasoning through Hypothesis Decomposition and Amendment (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.