Research map / Large language models / Reasoning and chain of thought

Learning to Reason Efficiently with Discounted Reinforcement Learning

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Efficient and adaptive thinking

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows via Blackwell optimality that discounting only the correctness reward in RLVR (GRPO) yields shortest-successful-path policies, and empirically shortens reasoning traces while matching Pass@1.

Problem

Large reasoning models generate excessive tokens, inflating cost and latency; it is unclear whether length reduction trades off against accuracy, and how to choose a principled length-penalty mechanism for RLVR.

Approach

Apply a discount factor gamma<1 to the verifier (correctness) reward as a function of reasoning-token count, within a finite restricted policy class; Blackwell optimal policies then maximize undiscounted accuracy and, among accuracy maximizers, minimize expected successful path length (SSP view). Train with discounted GRPO plus KL to a periodically refreshed reference policy, discounting only reas

Key results

• On GSM8K, discounted GRPO cuts mean response length by about 22% for Qwen2.5 7B-Instruct (217.60 to 170.08 tokens) with Pass@1 essentially unchanged (91.06 vs 91.07).
• On AIME 2025, discounted Phi-4 raises Pass@1 from 14.00 to 19.33 while reducing length from 1263.87 to 800.09 tokens.
• On MATH, discounted Qwen2.5 7B-Instruct reduces length from 491.32 to 384.96 tokens with Pass@1 64.55 vs 64.80 undiscounted.

Surprising findings

• Discounted training sometimes exceeds undiscounted accuracy (e.g., AMC 2023 for Phi-4: 51.00 to 61.00) while producing shorter responses.

Limitations

• Theory is restricted to finite policy classes; the main guarantees do not directly cover full softmax policies.
• Discount factor is chosen by bisection on training accuracy and tuned per model/dataset, and reported results are selected configurations.
• Some differences (e.g., AIME 2025 on small test sets) are within noise given few problems.

Experimental setup

Datasets: GSM8K MATH DeepScaleR

Benchmarks: GSM8K MATH AMC 2023 AIME 2025 MINERVA OLYMPIAD

Metrics: Pass@1 mean response length

Baselines: undiscounted GRPO VERL official baselines

Builds on: GRPO RLVR Blackwell optimality

Closest papers by meaning

• Thinking-Free Policy Initialization Makes Distilled Reasoning Models More Effective and Efficient Reasoners (ICLR 2026 · similarity 0.90)
• Promoting Efficient Reasoning with Verifiable Stepwise Reward (AAAI 2026 · similarity 0.89)
• Learn to Reason Efficiently with Adaptive Length-based Reward Shaping (ICLR 2026 · similarity 0.89)
• Stop Unnecessary Reflection: Training LRMs for Efficient Reasoning with Adaptive Reflection and Length Coordinated Penalty (ICLR 2026 · similarity 0.89)
• How Far Are We from Optimal Reasoning Efficiency? (NeurIPS 2025 · similarity 0.88)
• Learning to Reason under Off-Policy Guidance (NeurIPS 2025 · similarity 0.88)