Research map / Reinforcement learning / RL from feedback

AEM: Adaptive Entropy Modulation for Multi-Turn Agentic Reinforcement Learning

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: RL for LLM reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AEM, a supervision-free plug-in that rescales response-level advantages by a group-normalized response entropy proxy, derived from a response-level entropy-dynamics analysis, to improve multi-turn agentic RL.

Problem

Sparse outcome-only rewards in multi-turn LLM agent RL make step-level credit assignment ambiguous; existing fixes need process reward models, tree rollouts, or structural assumptions.

Approach

Lifts entropy dynamics from token to response level, shows entropy drift is governed by advantage times relative surprisal, and uses a group-normalized, length-averaged response entropy to upweight low-entropy and downweight high-entropy responses, giving an adaptive exploration-to-exploitation transition.

Key results

• GRPO+AEM with Qwen2.5-1.5B improves ALFWorld from 68.0% to 76.8% (+8.8%) and WebShop success from 65.0% to 70.6%.
• DeepSWE+AEM on Qwen3-32B reaches 43.7% resolved on SWE-bench-Verified vs 42.3% for DeepSWE (+1.4%).
• DAPO+AEM with Qwen2.5-7B reaches 96.6% on ALFWorld vs 96.1% for DAPO, with WebShop success improving from 86.7% to 88.9%.

Surprising findings

• Reversing the entropy-to-credit mapping (AEM_reverse) drops WebShop score to 77.2, below the GRPO baseline of 83.6.
• Shuffling the modulation coefficients within groups keeps performance near GRPO, indicating alignment between response and its entropy matters.

Limitations

• Response-level relative surprisal is approximated by a group-based, length-normalized entropy proxy rather than computed exactly.
• Improvements on several ALFWorld/WebShop settings are within reported standard deviations.
• Gains on SWE-bench-Verified are modest (+1.4%) and evaluated on a single 32B setting.

Experimental setup

Datasets: ALFWorld WebShop R2E

Benchmarks: ALFWorld WebShop SWE-bench-Verified

Metrics: success rate score resolved rate

Baselines: PPO GRPO GSPO DAPO ReAct GPT-5.2-Pro Gemini-3-Pro DeepSWE

Builds on: GRPO DAPO GSPO DeepSWE Fisher-Rao natural gradient

Closest papers by meaning

• Agentic Reinforced Policy Optimization (ICLR 2026 · similarity 0.90)
• LLM-Explorer: A Plug-in Reinforcement Learning Policy Exploration Enhancement Driven by Large Language Models (NeurIPS 2025 · similarity 0.88)
• Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents (NeurIPS 2026 · similarity 0.88)
• Learning Agentic Policy from Action Guidance (NeurIPS 2026 · similarity 0.88)
• Reasoning with Exploration: An Entropy Perspective (AAAI 2026 · similarity 0.88)
• SCOPE-RL: Stable and Quantitative Control of Policy Entropy in RL Post-Training (NeurIPS 2026 · similarity 0.88)