Research map / Large language models / Reasoning and chain of thought
Incentivizing LLM Reasoning via Reinforcement Learning with Functional Monte Carlo Tree Search
Kongcheng Zhang, QI YAO, Baisheng Lai, Jiaxing Huang, Wenkai Fang, Dacheng Tao, Mingli Song, Shunyu Liu
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Reinforced Functional Token Tuning (RFTT), which adds learnable functional tokens such as <analyze>, <verify>, and <refine> to the LLM vocabulary and uses reinforcement learning with Monte Carlo Tree Search to learn reasoning.
Problem
LLMs need learn-to-reason capabilities, and prior reasoning methods rely on prompting rather than learning the reasoning structure.
Approach
Embeds a set of learnable functional tokens directly into the model vocabulary so the model constructs chain-of-thought through these tokens, with reinforcement fine-tuning guided by functional Monte Carlo Tree Search.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train with RFTT (SFT warmup + RL with functional tree search) | Qwen-2.5-7B-Instruct · 5 math benchmarks average | Pass@1: 66.34 -> 75.66 | Improved |
| train with RFTT instead of ReFT | Qwen-3-4B-Base · 5 math benchmarks average | Pass@1: 76.22 -> 79.46 | Improved |
| train with RFTT instead of TreeRL | LLaMA-3.1-8B-Instruct · MATH-500 | Pass@1: 57.8 -> 60.2 | Improved |
| add RL phase after SFT warmup (ablation) | Qwen-2.5-7B-Instruct · 5 math benchmarks average | Pass@1: 69.86 -> 75.66 | Improved |
| add SFT warmup before RL (ablation) | Qwen-2.5-7B-Instruct · 5 math benchmarks average | Pass@1: 69.40 -> 75.66 | Improved |
| add functional-token MCTS sampling in RL instead of random sampling (ablation) | Qwen-2.5-7B-Instruct · 5 math benchmarks average | Pass@1: 72.34 -> 75.66 | Improved |
| add process reward model in tree search (ablation) | LLaMA-3.1-8B-Instruct · 5 math benchmarks average | Pass@1: 62.90 -> 64.88 | Improved |
| keep <verify> token available (ablation) | Qwen-2.5-7B-Instruct · MATH-500 | accuracy: 72.8% -> 79.8% | Improved |
| keep <subquestion> token available (ablation) | Qwen-2.5-7B-Instruct · MATH-500 | accuracy: 79.4% -> 79.8% | No clear effect |
| add MCTS in RL phase | RFTT training · training cost | time per step: 3.2 -> 12.1 min/step | Hurt |
| use RFTT trained on math only on out-of-domain tasks | Qwen-2.5-7B-Instruct · GPQA | accuracy: 36.4 -> 35.6 | No clear effect |
| combine RFTT with DAPO | Qwen-3-4B-Base · 5 math benchmarks average | Pass@1: 79.46 -> 80.78 | Improved |
Closest papers by meaning
- Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoning (NeurIPS 2025 · similarity 0.91)
- Native Reasoning Models: Training Language Models to Reason on Unverifiable Data (ICLR 2026 · similarity 0.91)
- Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective (NeurIPS 2025 · similarity 0.91)
- $\textbf{Re}^{2}$: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving (ICLR 2026 · similarity 0.91)
- Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Search (ICML 2025 · similarity 0.91)
- RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs (ICLR 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
