Research map / Large language models / Reasoning and chain of thought

Rethinking Thinking Tokens: LLMs as Improvement Operators

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Efficient and adaptive thinking · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes treating LLM inference as round-wise improvement operators (Sequential Refinement and Parallel-Distill-Refine with bounded summaries) and an operator-consistent RL objective, showing better AIME accuracy at matched sequential token budgets.

Problem

Long chain-of-thought reasoning raises accuracy but inflates context length, token cost, and latency; the paper asks whether short-context iterative refinement with compact summaries can give better accuracy-latency trade-offs.

Approach

Inference is cast as a read-write-compress cycle: parallel drafts are distilled into a bounded, non-persistent summary that conditions the next round, and RL training unrolls this same generate-distill-refine interface (mixed 1:1 with standard long-trace RL) to remove the train-test mismatch.

Key results

• For o3-mini at ~49k sequential tokens (16k thinking per call), accuracy on AIME 2024 rises from 76.9 (Long CoT) to 81.5 (SR) and 86.7 (PDR).
• At equal total budget of 442k tokens, PDR reaches the same accuracy as SR with a 2.57x smaller sequential budget (172k vs 442k).
• Operator RL continued from the baseline RL checkpoint gives +5.00 on AIME 2024 and +4.59 on AIME 2025 for the 8B model under PDR inference.

Surprising findings

• Admitting only incorrect candidates into the summary causes large accuracy drops, showing strong anchoring bias, especially for o3-mini.
• Error-analysis variant of SR helps o3-mini but not gemini-2.5-flash.

Limitations

• Evaluation limited to math competition benchmarks (AIME 2024/2025) and a small number of models.
• Operator RL trained with only one-round rollouts; multi-round inference uses the same operator without training for it.
• Gains from the operator RL are measured on a single 8B model and reported without error bars across seeds.

Experimental setup

Datasets: Polaris-53K DeepCoder AIME 2024 AIME 2025 MATH-500

Benchmarks: AIME 2024 AIME 2025

Metrics: mean@16 accuracy sequential token budget total token budget

Baselines: Long CoT o3-mini gemini-2.5-flash Baseline RL (CISPO) Parallel-R1 Self-consistency

Builds on: CISPO GRPO Self-Refine Compute as Teacher Parallel-R1

Closest papers by meaning

• Don’t Think Longer, Think Wisely: Optimizing Thinking Dynamics for Large Reasoning Models (NeurIPS 2025 · similarity 0.92)
• Thinker: Learning to Think Fast and Slow (NeurIPS 2025 · similarity 0.91)
• Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens (ICML 2026 · similarity 0.91)
• Rectifying LLM Thought from Lens of Optimization (ICLR 2026 · similarity 0.90)
• Think Thrice Before You Act: Progressive Thought Refinement in Large Language Models (ICLR 2025 · similarity 0.90)
• Towards Thinking-Optimal Scaling of Test-Time Compute for LLM Reasoning (NeurIPS 2025 · similarity 0.90)