Research map / Large language models / Reasoning and chain of thought

InftyThink+: Effective and Efficient Infinite-Horizon Reasoning via Reinforcement Learning

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Long CoT dynamics and efficiency · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes InftyThink+, an end-to-end RL framework that trains a model to decide when to summarize and how to resume iterative reasoning, using an SFT cold start followed by trajectory-level GRPO with task and efficiency rewards.

Problem

Long chain-of-thought reasoning suffers from quadratic inference cost, context-window limits, and lost-in-the-middle degradation; existing iterative (summarize-and-continue) reasoning relies on SFT or fixed heuristics and cannot optimize when to summarize, what to keep, and how to resume.

Approach

Extends InftyThink's model-controlled iterative summarization with reinforcement learning at the trajectory level: a multi-round rollout shares one trajectory-level advantage across all iterations, and the reward multiplies a task-correctness indicator by a quadratic iteration-efficiency penalty.

Key results

• On DeepSeek-R1-Distill-Qwen-1.5B, InftyThink+ (T) reaches 50.94% on AIME24 vs 38.75% for vanilla RL with task reward (+21.46 points over the cold-start InftyThink baseline's 29.48%).
• With task+efficiency reward, InftyThink+ reduces average inference latency to 48.37s from 149.44s for vanilla RL across the four benchmarks (about 67% lower, per the table averages).
• Reports 18.2% RL training speedup and 32.8% latency reduction on AIME25 versus vanilla approach.

Surprising findings

• Trajectory-level RL with task reward alone improves accuracy more than efficiency-aware RL, while adding efficiency reward cuts tokens and latency sharply at a modest accuracy cost relative to task-only RL.

Limitations

• Assumes reasoning decomposes into relatively independent stages; less useful for entangled reasoning.
• Natural-language summaries lack explicit control over information granularity and constraint strength.
• Depends on task-specific cold-start data construction.

Experimental setup

Datasets: vanilla long-CoT SFT data (transformed to InftyThink format) math RL datasets

Benchmarks: MATH500 AIME24 AIME25 GPQA_diamond

Metrics: accuracy (avg over 32 samples) average generated tokens inference latency

Baselines: Vanilla long CoT RL (GRPO) InftyThink (SFT) Markovian Thinker Delethink

Builds on: InftyThink GRPO DeepSeek-R1 IcePop Markovian Thinker

Closest papers by meaning

• Demystifying Long Chain-of-Thought Reasoning (ICML 2025 · similarity 0.89)
• ShorterBetter: Guiding Reasoning Models to Find Optimal Inference Length for Efficient Reasoning (NeurIPS 2025 · similarity 0.89)
• Thinking-Free Policy Initialization Makes Distilled Reasoning Models More Effective and Efficient Reasoners (ICLR 2026 · similarity 0.89)
• LoongRL: Reinforcement Learning for Advanced Reasoning over Long Contexts (ICLR 2026 · similarity 0.89)
• ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models (NeurIPS 2025 · similarity 0.89)
• DeepCompress: A Dual Reward Strategy for Dynamically Exploring and Compressing Reasoning Chains (ICLR 2026 · similarity 0.89)