Research map / Multi-agent systems and games / LLM agents

On Information Self-Locking in Reinforcement Learning for Active Reasoning of LLM agents

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Self-evolving language agent learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Identifies information self-locking (SeL) in outcome-based RL for active reasoning LLM agents, and proposes AReW, an advantage-reweighting method using directional critiques for action selection and belief tracking, improving PPO/GRPO/GSPO across multiple tasks.

Problem

Outcome-based RL for LLM agents performing active reasoning (must elicit observations through interaction) can get stuck in a low-information regime where agents neither elicit informative feedback nor internalize it.

Approach

Decomposes agent behavior into action selection (AS) and belief tracking (BT), theoretically shows their coupled learning bottleneck, and injects cheap binary/ternary directional critiques (from environment feedback and belief-readout changes) by additively reweighting per-step advantages, leaving the outcome reward unchanged.

Key results

• On PE-G S=3, PPO with AReW (AS+BT) on Qwen2.5-7B reaches 80.33 vs 18.33 vanilla PPO (up to 60+ point gains).
• AReW improves vanilla PPO in 27 of 28 evaluated settings across tasks and base models.
• On tau2-bench Telecom dual-control, average reward rises from 0.20 to 0.50 (2.5x) relative to vanilla PPO.

Surprising findings

• Replacing patient feedback with Unknown causes a smaller performance drop after RL training (61.00 to 55.50 vs 41.25 to 30.50 without RL), indicating reduced reliance on interaction.
• Group-based RL variants (GRPO, GSPO) still exhibit self-locking behavior, suggesting more rollouts alone do not resolve the AS-BT coupling.

Limitations

• Critiques rely on proxies (AS indicator, belief readout via prompting) that are instrumental and not exact belief states.
• Some tasks use rule-based or simulated users (GPT-4o / Qwen2.5-14B simulators) rather than real users.
• Theoretical assumptions are regularity conditions that are not verified in practice.

Experimental setup

Datasets: PE-G PE-F MediQ FloDial tau2-bench Telecom

Benchmarks: PE-G PE-F MediQ FloDial-Easy FloDial-Hard tau2-bench Telecom

Metrics: average outcome reward AS proxy BT proxy tool execution errors

Baselines: o4-mini direct inference PPO GRPO GSPO intermediate reward shaping

Builds on: PPO GRPO GSPO MediQ FloDial tau2-bench

Closest papers by meaning

• Reducing Belief Deviation in Reinforcement Learning for Active Reasoning (ICLR 2026 · similarity 0.88)
• Meta-Cognitive Memory Policy Optimization for Long-Horizon LLM Agents (NeurIPS 2026 · similarity 0.88)
• RAGEN-2: Reasoning Collapse in Agentic RL (ICML 2026 · similarity 0.88)
• MAGE: Multi-Agent Self-Evolution with Co-Evolutionary Knowledge Graphs (NeurIPS 2026 · similarity 0.88)
• Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents (NeurIPS 2026 · similarity 0.87)
• Unlocking the Power of Multi-Agent LLM for Reasoning: From Lazy Agents to Deliberation (ICLR 2026 · similarity 0.87)