Research map / Large language models / Reasoning and chain of thought

Thinking-Free Policy Initialization Makes Distilled Reasoning Models More Effective and Efficient Reasoners

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TFPI, a short multi-stage RLVR initialization stage that trains SFT-distilled reasoning models with thinking-free inputs (empty think tags) before standard DAPO RL, improving accuracy and token efficiency at lower compute.

Problem

RLVR on SFT-distilled long-CoT models needs very long training contexts, which is computationally expensive, while naive short-context multi-stage training degrades performance.

Approach

Append an empty think block (ThinkingFree operator) to queries during RLVR rollouts in short-context stages, so the policy learns to reason concisely; this cheap stage then initializes standard RLVR, improving slow-thinking accuracy and reducing tokens even when evaluated in thinking mode.

Key results

• Qwen3-4B TFPI+RL reaches 65.7 overall average vs 62.0 for Direct RL under matched compute.
• Qwen3-4B-2507 with TFPI only reaches 89.0% on AIME24 and 65.5% on LiveCodeBench using under 4K H20 hours.
• DS-1.5B thinking-free accuracy on AIME24 rises from 29.6% to 37.5% after TFPI stage 3 with about 5.3K vs 16.7K tokens.

Surprising findings

• RL trained with thinking-free rollouts at only 4K context improves thinking-mode AIME25 accuracy by about 2% while cutting output tokens by about 20%, whereas standard 4K RLVR drops avg@32 by over 40%.
• Thinking-free inference alone reduces output tokens by more than 70% for both DS-1.5B and Qwen3-4B on AIME25.

Limitations

• Training data is math-only (Polaris-53K); out-of-domain gains fluctuate across stages for some models.
• Gains in some configurations are modest relative to noise, and some baselines (e.g., Laser, AutoThink) are competitive on specific benchmarks.

Experimental setup

Datasets: Polaris-53K

Benchmarks: AIME24 AIME25 BeyondAIME GPQA-Diamond LiveCodeBench IFEval

Metrics: pass@1 (avg@k) average output tokens

Baselines: Direct RL Polaris-4B-Preview Polaris-7B-Preview DeepScaleR-1.5B Skywork-OR1-7B-Preview TLMRE AdaptThink AutoThink Laser L1-Max Thinkless

Builds on: DAPO GRPO PPO VeRL DeepScaleR

Closest papers by meaning

• Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement (ICML 2026 · similarity 0.91)
• $\textbf{Re}^{2}$: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving (ICLR 2026 · similarity 0.91)
• RLP: Reinforcement as a Pretraining Objective (ICLR 2026 · similarity 0.90)
• Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model (NeurIPS 2025 · similarity 0.90)
• Think or Not? Selective Reasoning via Reinforcement Learning for Vision-Language Models (NeurIPS 2025 · similarity 0.90)
• Learn to Reason Efficiently with Adaptive Length-based Reward Shaping (ICLR 2026 · similarity 0.90)