Research map / Large language models / Reasoning and chain of thought

Reasoning Cache: Continual Improvement Over Long Horizons via Short-Horizon RL

ICML 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Reasoning Cache (RC), an iterative summarize-then-reason decoding algorithm trained with outcome-reward RL and a summary replay buffer, so a 4B model can extrapolate reasoning far beyond its 16k training budget.

Problem

Standard RL trains LLM reasoning under a fixed prompt distribution and token budget, so models fail to extrapolate reasoning to much larger test-time budgets; autoregressive long rollouts also cause distribution shift and high training cost.

Approach

Replace autoregressive decoding during training and inference with an alternating loop that generates a bounded reasoning trace, summarizes it into a cache, and conditions the next turn only on the summary; train with per-turn outcome-reward GRPO on summary-conditioned generations, plus a replay buffer of past summaries to cover later turns cheaply.

Key results

• On HMMT 2025 (Nov), RCT-4B with Reasoning Cache decoding reaches 66.3% vs 39.8% for the base Qwen3-4B-Instruct-2507 at 16k, and nearly 70% at 0.5M test tokens per the abstract.
• RCT-4B + Reasoning Cache scores 58.0 on IMO-AnswerBench and 74.9 on AIME 2025, outperforming Qwen3-4B-Thinking-2507 (53.8 and 81.3 respectively on the AIME case being lower) on IMO-AnswerBench.
• Standard GRPO at 32k training budget gives only modest gains and no FrontierScience improvement (21.5 vs 23.3 base), while RCT-4B reaches 34.1 on FrontierScience.

Surprising findings

• Standard RL-trained model with the same data and curriculum decreases on FrontierScience relative to the base model, while iterative decoding methods (even untrained) outperform autoregressive decoding on FrontierScience.
• Summary-generation-only training hurts performance, leaving the model no better than base.

Limitations

• Training uses myopic per-turn rewards, discouraging multi-turn strategies where early turns explore without immediate reward.
• Relies on summarization-generation asymmetry, which is weak or absent in specialized reasoning models (e.g., Qwen3-4B-Thinking), and may not hold for search-heavy problems.
• Training only on summary-conditioned generation; summary generation is not optimized.

Experimental setup

Datasets: AceReason-Math DAPO Omni-MATH HMMT 2025 AIME 2025 IMO-AnswerBench FrontierScience

Benchmarks: AIME 2025 HMMT 2025 (Nov) IMO-AnswerBench FrontierScience

Metrics: accuracy pass@k

Baselines: Qwen3-4B-Instruct-2507 Qwen3-4B-Thinking-2507 Polaris-4B Qwen3-4B-Instruct-2507 (RL, 32k) Self-Refine Self-Verify

Builds on: GRPO Qwen3-4B-Instruct-2507 Recursive Self-Aggregation DeepSeekMath agent

Closest papers by meaning

• Deliberation in Latent Space via Differentiable Cache Augmentation (ICML 2025 · similarity 0.90)
• Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs (ICLR 2026 · similarity 0.89)
• InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory (NeurIPS 2024 · similarity 0.89)
• Training Large Reasoning Models Efficiently via Progressive Thought Encoding (ICLR 2026 · similarity 0.89)
• MILES: Modular Instruction Memory with Learnable Selection for Self-Improving LLM Reasoning (NeurIPS 2026 · similarity 0.89)
• You Only Need Minimal RLVR Training: Extrapolating LLMs via Rank-1 Trajectories (NeurIPS 2026 · similarity 0.88)