Research map / Large language models / Reasoning and chain of thought

DeepCompress: A Dual Reward Strategy for Dynamically Exploring and Compressing Reasoning Chains

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DeepCompress, a zero-RL method with a dual length reward (favor short responses for "Simple" questions, long for "Hard") where difficulty is judged in real time from group vs. smoothed batch pass ratio, improving both accuracy and token efficiency on math benchmarks.

Problem

Length-reduction RL methods for large reasoning models improve efficiency but often cost accuracy, since shortening responses can shrink the set of correct solutions found for hard problems.

Approach

Adaptive, model-aware length reward within GRPO/DAPO: the sign of beta = (group pass ratio - smoothed batch pass ratio) decides whether a question's length reward favors shorter (Simple) or longer (Hard) standardized responses, with the length reward applied only to correct responses and the batch ratio smoothed by EMA.

Key results

• DeepCompress-Zero-7B reaches 48.7 average accuracy across seven math benchmarks versus 46.0 for DeepMath-Zero-7B (+2.7), with +4.1 on AIME24 and +6.5 on AIME25.
• DeepCompress-Zero-3B averages 36.6 versus 34.6 for DeepMath-Zero-3B (+2.0 points).
• DeepCompress reduces average response length by 57.9% at 3B and 16.6% at 7B relative to DeepMath-Zero.

Surprising findings

• Longer responses cover more correct solutions at pass@k even though shorter responses win at pass@1, motivating the reverse length preference for hard problems.
• A fixed length bonus (beta=-1) yields higher performance than the adaptive method on some measures but at the cost of much longer outputs and higher entropy.

Limitations

• Effectiveness depends on sufficient length variation among responses sampled within an RL group.
• Maximum generation length capped at 10K tokens during training, which may limit exploration of longer solutions.
• Evaluation focuses on math benchmarks with 3B and 7B Qwen2.5 models only.

Experimental setup

Datasets: DeepMath-103K MATH AIME 2022-2023 (validation)

Benchmarks: MATH-500 AMC 2023 OlympiadBench Minerva Math AIME 2024 AIME 2025 PolyMath (English) GPQA-Diamond MMLU-STEM Big-Bench Hard

Metrics: pass@1 average response length pass@k

Baselines: DeepMath-Zero-3B DeepMath-Zero-7B Open-Reasoner-Zero-7B Qwen-2.5-7B-SRL-Zoo Qwen-2.5-3B-Instruct Length Penalty Length Bonus

Builds on: DAPO GRPO DeepMath-Zero

Closest papers by meaning

• Learn to Reason Efficiently with Adaptive Length-based Reward Shaping (ICLR 2026 · similarity 0.91)
• DeepSearch: Overcome the Bottleneck of Reinforcement Learning with Verifiable Rewards via Monte Carlo Tree Search (ICLR 2026 · similarity 0.90)
• Making Slow Thinking Faster: Compressing LLM Chain-of-Thought via Step Entropy (ICLR 2026 · similarity 0.90)
• ShorterBetter: Guiding Reasoning Models to Find Optimal Inference Length for Efficient Reasoning (NeurIPS 2025 · similarity 0.90)
• Don’t Think Longer, Think Wisely: Optimizing Thinking Dynamics for Large Reasoning Models (NeurIPS 2025 · similarity 0.90)
• Promoting Efficient Reasoning with Verifiable Stepwise Reward (AAAI 2026 · similarity 0.90)