Research map / Large language models / Reasoning and chain of thought

Rectifying LLM Thought from Lens of Optimization

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Chain-of-thought training and analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RePro, a plug-and-play process-level reward that scores CoT segments by the intensity and stability of a log-likelihood-of-ground-truth surrogate objective, integrated into RLVR to improve reasoning accuracy and token efficiency.

Problem

Long-CoT LLMs often overthink and produce protracted, suboptimal reasoning chains that hurt accuracy and cost tokens; outcome-only RLVR does not shape the intermediate reasoning process.

Approach

Frames CoT as a gradient-descent-like optimization whose surrogate objective is the mean log-probability of the ground-truth answer appended after each partial trajectory; a dual score (tanh-normalized magnitude gain plus Kendall-tau stability) is computed on entropy-selected segments and used as a group-normalized process reward added to the outcome advantage.

Key results

• On DeepSeek-R1-Distill-Qwen-1.5B with GRPO, RePro raises AIME24 from 32.9 to 36.0 and LiveCodeBench from 15.2 to 18.4.
• With PPO on DeepSeek-R1-Distill-Qwen-1.5B, AIME24 improves from 34.8 to 36.3 and MATH500 from 86.9 to 87.7.
• Compared with vanilla GRPO, the method reduces average tokens from 12,367 to 6,558 on the reported benchmarks while improving accuracy (e.g., +3.1 on AIME24).

Surprising findings

• On zero-RLVR training of Qwen3-4B-Base, the method is comparable to GRPO and slightly lower on AIME24 (21.0 vs 23.5).
• On Qwen3-1.7B with PPO, a few benchmarks such as LMB and GPQA-D show no gain or small drops, and Hunyuan-1.8B shows lower AIME25 and MATH500 with RePro in some pairings.

Limitations

• Surrogate objective assumes the ground-truth answer log-probability is a valid proxy for optimization progress; the paper provides only a small empirical illustration on one question.
• Hyperparameters w, alpha, and k were tuned on a limited set of backbones, and gains are often small (about 1 point) with unreported variance.
• Relies on an external LLM (Qwen3-235B) for thinking-pattern analysis, and the process-reward computation adds overhead.

Experimental setup

Datasets: DeepScaleR-Preview-Dataset

Benchmarks: AIME24 AIME25 MATH500 LiveMathBench GPQA-Diamond MBPP LiveCodeBench

Metrics: Avg@16 accuracy Avg@4 accuracy average token count

Baselines: PPO REINFORCE++ REINFORCE++ Baseline GRPO ThinkPrune L1 Laser AdaThink

Builds on: PPO GRPO REINFORCE++ DeepScaleR

Closest papers by meaning

• Demystifying Long Chain-of-Thought Reasoning (ICML 2025 · similarity 0.93)
• When More is Less: Understanding Chain-of-Thought Length in LLMs (ICLR 2026 · similarity 0.93)
• $\textbf{Re}^{2}$: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving (ICLR 2026 · similarity 0.92)
• Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs (ICLR 2026 · similarity 0.92)
• Well Begun, Half Done: Reinforcement Learning with Prefix Optimization for LLM Reasoning (AAAI 2026 · similarity 0.92)
• Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Search (ICML 2025 · similarity 0.92)