Research map / Reinforcement learning / MDPs and planning

When Can Model-Free Reinforcement Learning be Enough for Thinking?

NeurIPS 2025 · Reinforcement learning › MDPs and planning · approach: POMDPs and confounded RL · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces the thought MDP formalism to explain when model-free RL yields "thinking" actions, proves policy-initialization conditions for emergence, probes LLMs for thinking benefits, and shows a gridworld where multi-task pretraining plus thought actions speeds REINFORCE.

Problem

Under what conditions does model-free RL produce thinking actions, which neither yield reward nor change the external state, as a reward-maximizing strategy?

Approach

Extends MDPs with thought states and thought actions that alter only the agent's internal state; shows thought actions equal a policy-improvement step, so thinking emerges when the initial policy has useful sub-policies reachable via thought actions. Argues LLM chain-of-thought fits this model and that pretraining plus thought actions helps RL in a non-language gridworld.

Key results

• Appending forced partial-sum thinking tokens raised accuracy on five-term four-digit addition for all 11 open models, e.g., Qwen2.5-7B-Instruct from 5.06% to 96.10% and Llama-2-7b from 0.00% to 48.80%.
• In the gridworld, Pretrained-Think learns significantly faster than Pretrained-NoThink, and from-scratch agents fail to learn with or without thinking under sparse reward.
• Thought actions occur about 30% of the time early in training and settle around 15%, close to the optimal 2 thinking steps per 14-step episode.

Measured effects reported in the paper

Surprising findings

• Thought actions persisted at about 15% of steps at convergence even though the theory predicts the optimal policy never thinks; the authors attribute this to sampling error in on-policy policy gradient.
• Pretrained-NoThink also began learning, suggesting pretraining helps even without thought actions because environment action sequences can trigger sub-task behavior.

Limitations

• Theory assumes deterministic thought transitions, non-negative rewards, and a static environment during thinking.
• The LLM evidence is correlational: it forces thinking tokens rather than learning them with RL, so it only shows the precondition of increased value.
• The non-language experiment is a single small toy gridworld with hand-designed sub-tasks.

Experimental setup

Datasets: gridworld rollouts generated by a scripted agent 1000 random four-integer addition prompts per model condition

Benchmarks: 5x5 gridworld navigation task (bottom-right then top-left) five-term four-digit addition

Metrics: task success rate response accuracy with 95% confidence interval thinking time fraction

Baselines: Scratch-Think Scratch-NoThink Pretrained-NoThink

Builds on: Markov decision process generalized policy iteration effective horizon (Jiang et al.) options framework zero-shot chain-of-thought

Closest papers by meaning

• Think or Not? Selective Reasoning via Reinforcement Learning for Vision-Language Models (NeurIPS 2025 · similarity 0.89)
• Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Models (NeurIPS 2025 · similarity 0.89)
• Thinking-Free Policy Initialization Makes Distilled Reasoning Models More Effective and Efficient Reasoners (ICLR 2026 · similarity 0.89)
• All Roads Lead to Rome: Incentivizing Divergent Thinking in Vision-Language Models (CVPR 2026 · similarity 0.88)
• Hypothetical Minds: Scaffolding Theory of Mind for Multi-Agent Tasks with Large Language Models (ICLR 2025 · similarity 0.88)
• Interpreting Emergent Planning in Model-Free Reinforcement Learning (ICLR 2025 · similarity 0.88)