Research map / Large language models / Reasoning and chain of thought

RAGEN-2: Reasoning Collapse in Agentic RL

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Efficient and adaptive thinking

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Identifies "template collapse" in multi-turn LLM agent RL (input-agnostic reasoning hidden by stable entropy), proposes an in-batch cross-scoring mutual-information proxy to detect it, and an SNR-aware prompt filter based on reward variance to mitigate it.

Problem

Multi-turn LLM agent RL is unstable and reasoning quality degrades, but standard entropy monitors only measure within-input diversity and cannot detect reasoning that stops depending on the input.

Approach

Decomposes marginal reasoning entropy into conditional entropy and input-dependence mutual information, estimates the latter via in-batch cross-scoring of reasoning traces against all batch prompts, and explains collapse via a gradient SNR argument that motivates filtering low-reward-variance prompts before each update.

Key results

• SNR-Aware Filtering improves average success over the PPO baseline on the 4-task matrix (67.6% to 74.5% average gain of +6.9 points on the PPO row; GRPO row +3.7 average).
• MI-ZScore (trajectory) reaches Spearman +0.39 with task performance, versus -0.11 to -0.14 for entropy metrics.
• On Sokoban, prompt-level RV filtering at rho=0.9 raises task performance from 12.9% to 23.6% (quartile/filter analysis) and improves MI proxy from 0.83 to 1.80.

Surprising findings

• Conditional entropy stays elevated while input dependence declines, so entropy-based monitors point in the wrong direction for performance.
• Format validity is largely decoupled from collapse: runs keep near-perfect format validity while MI is low.

Limitations

• SNR decomposition assumes task signal and regularization noise separate cleanly, which may not hold under gradient accumulation.
• Reward variance is unreliable in sparse or noisy reward environments, and the kept mass requires per-task tuning.
• All experiments are single-agent; multi-agent propagation is untested.

Experimental setup

Datasets: MetaMathQA Countdown DeepCoder SearchQA WebShop Sokoban FrozenLake

Benchmarks: Sokoban FrozenLake MetaMathQA Countdown SearchQA WebShop DeepCoder

Metrics: success rate Spearman correlation retrieval accuracy conditional entropy

Baselines: PPO DAPO GRPO Dr. GRPO top-k filtering no filtering entropy regularization KL regularization

Builds on: PPO GRPO DAPO Dr. GRPO RAGEN

Closest papers by meaning

• STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stability (NeurIPS 2026 · similarity 0.88)
• Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning (NeurIPS 2025 · similarity 0.88)
• Reasoning with Exploration: An Entropy Perspective (AAAI 2026 · similarity 0.88)
• Mitigating Overthinking in Large Reasoning Language Models via Reasoning Path Deviation Monitoring (NeurIPS 2026 · similarity 0.88)
• ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory (ICLR 2026 · similarity 0.88)
• Recognize Your Orchestrator: An Entropy Dynamics Perspective for LLM Multi-Agent Systems (ICML 2026 · similarity 0.88)