Research map / Large language models / Reasoning and chain of thought

The Overthinker's DIET: Cutting Token Calories with DIfficulty-AwarE Training

NeurIPS 2025 · Large language models › Reasoning and chain of thought · approach: Test-time compute scaling · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DIET, a difficulty-aware RL framework for compressing LLM reasoning length (adaptive penalty strength plus dynamic length targets), with an Advantage Weighting fix for GRPO normalization, applied to R1-Distill-Qwen-1.5B.

Problem

Reasoning LLMs overthink, producing very long responses even for easy problems; existing length-penalty and compression methods use a uniform penalty that ignores problem difficulty and often degrades accuracy or disrupts the natural length-difficulty correlation.

Approach

Estimate per-prompt difficulty on the fly from GRPO group correctness, scale the length penalty by estimated easiness, condition penalties on difficulty-dependent target lengths, and normalize outcome and penalty advantages separately (Advantage Weighting) so the penalty weight is not distorted by outcome variance; add cyclical penalty modulation.

Key results

• DIET reaches a macro-average Pass@1 of 50.2% versus 48.6% for the base model while cutting average tokens by 40.7% (10280 to 6097).
• Kimi 1.5 RL and O1-Pruner reduce tokens by about 60% and 53% but lose 27.9% and 5.4% relative macro-average accuracy in the paper's table, while TokenSkip reduces tokens by 83.5% but drops macro P@1 to 30.9%.
• Dynamic Target alone keeps macro-average Pass@1 at 48.6% with 45.0% fewer tokens; Adaptive Weighting alone reaches 49.9% with 44.3% fewer tokens.

Measured effects reported in the paper

Surprising findings

• Naive reward weighting before GRPO normalization makes the full DIET configuration fail to deliver reasonable performance, per the paper.
• Kimi 1.5 SFT reduces accuracy by 12.1% while cutting tokens only 4.0%, showing SFT-based compression can hurt more than it saves.

Limitations

• Evaluation is limited to mathematical reasoning benchmarks and one 1.5B base model.
• Code and data are not yet released; the authors promise to open-source after organization.
• The Advantage Weighting derivation assumes outcome and penalty are approximately independent within a batch.

Experimental setup

Datasets: DeepScaleR

Benchmarks: MATH 500 AIME 2024 AMC 2023 Olympiad Bench Minerva

Metrics: Pass@1 average response tokens majority voting accuracy Pearson correlation between difficulty and length

Baselines: Kimi 1.5 SFT Kimi 1.5 DPO TokenSkip CosFn O1-Pruner Kimi 1.5 RL Base Model

Builds on: GRPO Kimi 1.5 DeepScaleR veRL TokenSkip

Closest papers by meaning

• $\textbf{Re}^{2}$: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving (ICLR 2026 · similarity 0.90)
• T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling (ICML 2025 · similarity 0.89)
• LEAD: Length-Efficient Adaptive and Dynamic Reasoning for Large Language Models (NeurIPS 2026 · similarity 0.89)
• DiffAdapt: Difficulty-Adaptive Reasoning for Token-Efficient LLM Inference (ICLR 2026 · similarity 0.89)
• DeepCompress: A Dual Reward Strategy for Dynamically Exploring and Compressing Reasoning Chains (ICLR 2026 · similarity 0.89)
• Advancing LLM Reasoning with Natural Language and Numerical Feedback (ICML 2026 · similarity 0.89)