Research map / Large language models / Reasoning and chain of thought

RL of Thoughts: Navigating LLM Reasoning with Inference-time Reinforcement Learning

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Trains a tiny (<3K-parameter) RL navigator with Double-Dueling-DQN and a PRM reward to dynamically select and compose five cognition-inspired logic blocks at inference time, improving frozen LLM reasoning across math, STEM and commonsense benchmarks.

Problem

Inference-time reasoning techniques (CoT, ToT, GoT) use manually predefined, task-agnostic logical structures that cannot adapt to diverse tasks or to the changing state of multi-step reasoning, while fine-tuning for reasoning is compute-expensive.

Approach

Model LLM reasoning as an MDP whose actions are five basic logic blocks (reason one step, decompose, debate, refine, terminate), with states from LLM self-evaluation and rewards from a process reward model; a small MLP navigator is trained with RL while the LLM and PRM stay frozen, so the navigator composes task-specific logical structures at inference time.

Key results

• RLoT with Qwen2.5-14B-Instruct averages 69.19 across seven benchmarks versus 64.57 for the best single-round baseline (Zero-shot CoT) and 62.94 for CoT-SC.
• With Llama3.1-8B-Instruct, RLoT improves GPQA from 33.48 (CoT-SC) to 46.88, a gain the paper reports as 13.4%.
• A navigator with 2,566 parameters lifts sub-10B models toward larger ones; RLoT on GPT-4o-mini averages 79.23 vs 78.59 for the 200B-scale few-shot CoT reference.

Surprising findings

• ToT performs worse than simpler baselines such as zero-shot CoT and CoT-SC on many tasks.
• A navigator trained with one LLM transfers to other LLMs and tasks without fine-tuning, with performance roughly consistent across cross-tests.

Limitations

• Navigator decisions depend on LLM self-evaluation, which can hallucinate correctness and mislead action selection.
• Action selection errors can cause unnecessary refinement that overwrites correct steps.
• Parameter counts for GPT-4o series are estimated in prior work, so size comparisons are approximate.

Experimental setup

Datasets: MATH GPQA StrategyQA

Benchmarks: AIME24 AMC23 MATH GSM8K GPQA MMLU-STEM StrategyQA MATH-500

Metrics: accuracy token consumption solving time

Baselines: Direct QA Zero-shot CoT Few-shot CoT CoT-SC ToT Self-Refine Q* rStar AFlow Buffer of Thoughts

Builds on: Chain-of-Thought Tree-of-Thoughts Graph-of-Thoughts Double DQN Dueling network Math-Shepherd

Closest papers by meaning

• RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs (ICLR 2026 · similarity 0.93)
• Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.93)
• Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective (NeurIPS 2025 · similarity 0.93)
• On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models (ICML 2026 · similarity 0.92)
• RAST: Reasoning Activation in LLMs via Small-model Transfer (NeurIPS 2025 · similarity 0.92)
• Beyond Accuracy: Dissecting Mathematical Reasoning for LLMs Under Reinforcement Learning (NeurIPS 2025 · similarity 0.92)