Research map / Large language models / Reasoning and chain of thought

Overthinking Reduction with Decoupled Rewards and Curriculum Data Scheduling

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DeCS, which combines a decoupled token-level length reward (penalizing only tokens after the necessary reasoning prefix, detected by a small judge model) with a curriculum batch schedule, cutting reasoning tokens by about 50% on DeepSeek-R1-Distill models while keeping accuracy.

Problem

Length penalties added to GRPO-style RLVR reduce overthinking in large reasoning models but often degrade accuracy and leave redundant reasoning, due to misalignment between trajectory-level rewards and token-level updates.

Approach

Identify the necessary reasoning prefix (NRP) with a judge model, assign high reward to NRP tokens and a length-scaled lower reward to tokens after it, and schedule the fraction of easy prompts per batch according to the NRP ratio so that correct high-entropy exploratory tokens are not over-suppressed.

Key results

• On DS-1.5B, DeCS reduces average reasoning tokens by 57.17% while improving average pass@1 by +2.48 points (AES 0.74 vs 0.62 for the best baseline ThinkPrune).
• On DS-7B, DeCS cuts thinking tokens by 49.50% with a +0.8 point accuracy gain over the base model.
• On Qwen3-4B, DeCS reduces reasoning tokens by 54.80% with 1.32 pass@1 improvement on average.

Surprising findings

• Pass@k curves for the compressed model almost overlap with the base model across k up to 128, suggesting exploration capacity is preserved.
• Removing the decoupled reward leaves roughly 25% overthinking tokens, indicating sequence-level length rewards under-reduce redundancy.

Limitations

• NRP detector is specialized for math reasoning and trained on math data; the authors evaluate models only up to 7B (4B for Qwen3).
• Adds a separate judge model and an NRP detection step, with roughly 3-5% training time overhead.
• The theoretical analysis relies on simplifying assumptions (tabular softmax policy, i.i.d. outputs, uniform high-entropy token distribution).

Experimental setup

Datasets: DeepScaleR MATH500 AMC23 OlympiadBench AIME2024 AIME2025 GPQA-Diamond LiveCodeBench-v6

Benchmarks: AIME2024 AIME2025 AMC23 MATH500 OlympiadBench GPQA-Diamond LiveCodeBench-v6

Metrics: pass@1 average tokens average efficiency score (AES) pass@k

Baselines: GRPO ThinkPrune TLMRE AdaptThink LC-R1

Builds on: GRPO DAPO veRL AdaptThink DeepScaleR

Closest papers by meaning

• Promoting Efficient Reasoning with Verifiable Stepwise Reward (AAAI 2026 · similarity 0.90)
• Learn to Reason Efficiently with Adaptive Length-based Reward Shaping (ICLR 2026 · similarity 0.89)
• Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoning (ICLR 2026 · similarity 0.88)
• h1: Bootstrapping LLMs to Reason over Longer Horizons via Reinforcement Learning (ICML 2026 · similarity 0.88)
• Scheduling Your LLM Reinforcement Learning with Reasoning Trees (ICLR 2026 · similarity 0.87)
• CurES: From Gradient Analysis to Efficient Curriculum Learning for Reasoning LLMs (ICLR 2026 · similarity 0.87)