Research map / Large language models / Reasoning and chain of thought

Training Large Reasoning Models Efficiently via Progressive Thought Encoding

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Progressive Thought Encoding, a LoRA-based method that compresses tokens evicted from a sliding-window KV cache into fixed-size latent adapter updates so LRMs can be RL-trained and run under tight cache budgets.

Problem

RL training of large reasoning models requires long rollouts whose autoregressive decoding dominates memory and time, while sliding-window KV cache eviction bounds memory but degrades long-context reasoning.

Approach

Instead of discarding evicted thinking tokens, encode them into a fixed-size context state S_e using learnable global tokens and a global query, and fold this state into LoRA weights online, so memory stays constant without backpropagating through full-cache rollouts.

Key results

• Averaged over benchmarks, the 3B model reaches 30.1% vs 28.2% for LoRA and 26.9% for baseline, and the DeepSeek-R1-Distill-Llama-8B reaches 45.6% vs 34.9% for LoRA.
• On AIME2025 with DeepSeek-R1-Distill-Llama-8B, the method reaches 43.3 pass@16 vs 20.0 for LoRA and 20.0 for baseline.
• Across cache budgets, the method averages 39.37 vs 32.99 for LoRA (+19.3% relative) and 30.31 for baseline (+29.9% relative).

Surprising findings

• Reported that the method with a constrained cache can outperform full-cache LoRA training on several benchmarks, e.g., 3B model average 30.1 vs 28.2.
• Advanced token-dropping methods such as HeadKV gave only +2.3% accuracy for +37% rollout runtime, so the authors kept sliding-window eviction.

Limitations

• Training uses only a sliding-window eviction strategy; more sophisticated importance-based eviction adds overhead and was not used in training.
• Evaluated only on math reasoning benchmarks.
• Advanced token dropping during rollouts increases runtime substantially (+37%).

Experimental setup

Datasets: DAPO-Math-17K

Benchmarks: MATH500 OlympiadBench Minerva Math AMC AIME2024 AIME2025

Metrics: pass@1 pass@16 peak GPU memory mean GPU memory attention TFLOPs

Baselines: LoRA LoRA with sliding-window cache (LoRA_c) Baseline (no fine-tuning) H2O PyramidKV HeadKV

Builds on: GRPO LoRA DAPO H2O PyramidKV HeadKV

Closest papers by meaning

• Reasoning Under 1 Billion: Memory-Augmented Reinforcement Learning for Large Language Models (NeurIPS 2026 · similarity 0.91)
• ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models (NeurIPS 2025 · similarity 0.91)
• Which Heads Matter for Reasoning? RL-Guided KV Cache Compression (ICML 2026 · similarity 0.90)
• ThinKV: Thought-Adaptive KV Cache Compression for Efficient Reasoning Models (ICLR 2026 · similarity 0.90)
• Learn to Reason Efficiently with Adaptive Length-based Reward Shaping (ICLR 2026 · similarity 0.90)
• RL of Thoughts: Navigating LLM Reasoning with Inference-time Reinforcement Learning (ICLR 2026 · similarity 0.90)