Research map / Large language models / Reasoning and chain of thought

Retrieval-of-Thought: Efficient Reasoning via Reusing Thoughts

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: KV cache compression

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Retrieval-of-Thought (RoT), which stores reasoning steps from ~3.34k math templates in a thought graph and uses reward-guided traversal to assemble query-specific templates inserted into Qwen3 prompts (and <think> tags via Thinking Intervention), reducing output tokens and latency on AIME/AMC.

Problem

Large reasoning models produce long reasoning traces, inflating latency and cost; existing retrieval-based template reuse (BoT, SuperCorrect, RAG) uses static templates that cannot be recombined for new problems.

Approach

Decompose reasoning templates into individual steps stored in a thought graph with sequential and semantic edges; at inference, retrieve an initial node by query similarity and greedily traverse the graph with a reward (query relevance + sequential flow) to assemble a problem-specific template that is inserted into the reasoning prompt (optionally inside think tags via Thinking Intervention).

Key results

• RoT+TI on Qwen3-1.7B reaches 44% accuracy vs 46% for CoT while reducing output length from 11.0k to 8.0k tokens (text-reported averages).
• Across models, RoT+TI reduces end-to-end latency by up to 82% and cost by 59.0% on Qwen3-0.6B relative to CoT.
• RoT+TI cuts path-switch markers by up to 81.8% on Qwen3-0.6B relative to CoT.

Surprising findings

• Standard RoT (template in prompt without think tags) often increases output tokens relative to CoT for several models, while only the think-tag variant (RoT+TI) reduces them.
• Accuracy gains and token savings are larger for smaller models; the authors attribute this to instruction-following retention after RL training in larger models.

Limitations

• Accuracy drops for some configurations (e.g., RoT+TI on Qwen3-0.6B falls to 0% on AIME 2024/2023), and results are mixed across models.
• Evaluation centers on math benchmarks with a single thought graph built from manually tagged templates; tagging is manual.
• Reported cost and accuracy numbers are on small benchmarks (30 AIME problems and 40 AMC problems), so variance is high.

Experimental setup

Datasets: ReasonFlux-v2 templates AIME 2023 AIME 2024 AIME 2025 AMC 2023 GPQA

Benchmarks: AIME 2025 AIME 2024 AIME 2023 AMC 2023 GPQA

Metrics: accuracy output tokens latency cost path switches GPU memory

Baselines: CoT CoT-SC RAG BoT

Builds on: Buffer of Thought Thinking Intervention Retrieval-Augmented Thoughts GRPO

Closest papers by meaning

• A*-Thought: Efficient Reasoning via Bidirectional Compression for Low-Resource Settings (NeurIPS 2025 · similarity 0.89)
• Reasoning or Retrieval? A Study of Answer Attribution on Large Reasoning Models (ICLR 2026 · similarity 0.89)
• Algorithm of Thoughts: Enhancing Exploration of Ideas in Large Language Models (ICML 2024 · similarity 0.88)
• Accelerating Large Language Model Reasoning via Speculative Search (ICML 2025 · similarity 0.88)
• Think or Not? Exploring Thinking Efficiency in Large Reasoning Models via an Information-Theoretic Lens (NeurIPS 2025 · similarity 0.88)
• The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity (NeurIPS 2025 · similarity 0.88)