Research map / Large language models / Reasoning and chain of thought

Training Language Models to Reason Efficiently

NeurIPS 2025 · Large language models › Reasoning and chain of thought · approach: Routing, pruning and distillation for reasoning · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a length-penalized RL objective (per-prompt normalized sigmoid length penalty on correct responses, optimized with PPO+RLOO) that trains DeepSeek-R1-Distill-Qwen 1.5B/7B to reason with fewer tokens, controllable via a single coefficient alpha.

Problem

Large reasoning models produce very long chain-of-thoughts, which makes inference expensive in cost, latency, and energy; the goal is to reduce token usage while preserving accuracy.

Approach

Reward correct responses with a length-penalty term, (1 - alpha * sigmoid((len - mean)/std)), where mean and std are per-prompt statistics over correct rollouts, and optimize it online with PPO using an RLOO advantage. A single alpha yields a family of models along the accuracy-token tradeoff curve.

Key results

• On the 7B model, alpha=0.1 cuts MATH500 tokens by about 36% (~4000 to ~2600) with a 2.2-point accuracy drop.
• On AIME 2024, alpha=0.2 reduces 7B tokens by 27% (~13,000 to ~9,000) with about a 4-point accuracy drop.
• On the 7B model, GSM8K tokens fall by up to 83% at alpha=0.2 with small accuracy loss; the 1.5B SFT and DPO baselines are less accurate at matched token usage.

Measured effects reported in the paper

Surprising findings

• Distilled reasoning models largely fail to follow explicit length instructions such as 'answer in less than X tokens', with generated length uncorrelated with the requested limit.
• Advantage normalization with the length penalty causes a length decrease independent of alpha and a sharp accuracy drop, and even alpha=0 yields shorter responses due to RLOO loss-length bias in OpenRLHF.

Limitations

• Optimizing alpha controls cost only indirectly and does not target a precise generation length.
• The length penalty incurs a small accuracy loss relative to the original reasoning model.
• The RL pipeline is more complex than SFT or DPO alternatives.

Experimental setup

Datasets: NuminaMath (MATH, cn_k12, AIME, AoPS, Olympiad subsets) MATH500 GSM8K AIME 2024 CommonSenseQA BIG-Bench Logical Deduction MMLU

Benchmarks: GSM8K MATH500 AIME 2024 CommonSenseQA Logical Deduction

Metrics: average pass rate@k average tokens generated normalized accuracy normalized tokens faithfulness score

Baselines: DeepSeek-R1-Distill-Qwen-1.5B DeepSeek-R1-Distill-Qwen-7B Generation Cutoff Rejection Sampling + SFT DPO O1-Pruner

Builds on: PPO RLOO DPO GRPO OpenRLHF

Closest papers by meaning

• Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes (ICLR 2026 · similarity 0.94)
• Large Language Model Cascades with Mixture of Thought Representations for Cost-Efficient Reasoning (ICLR 2024 · similarity 0.91)
• A State-Transition Framework for Efficient LLM Reasoning (ICLR 2026 · similarity 0.91)
• Improving Reasoning Performance in Large Language Models via Representation Engineering (ICLR 2025 · similarity 0.91)
• Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens (ICML 2026 · similarity 0.91)
• Reasoning-as-Logic-Units: Scaling Test-Time Reasoning in Large Language Models Through Logic Unit Alignment (ICML 2025 · similarity 0.91)