Research map / Multi-agent systems and games / LLM agents

Exploratory Memory-Augmented LLM Agent via Hybrid On- and Off-Policy Optimization

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: Self-evolving language agent learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes EMPO², a hybrid online RL method for LLM agents that mixes memory-augmented and memory-free rollouts, self-generated tips as memory, and on/off-policy updates (tips removed in off-policy updates), plus a novelty intrinsic reward.

Problem

LLM agents trained with RL under-explore in environments requiring discovery of novel states; GRPO converges prematurely on ScienceWorld and WebShop, and memory-based methods saturate because static parameters cannot capture needed diversity.

Approach

The policy itself generates reflective tips stored in an external memory; rollouts are split between memory-augmented and memory-free prompting, and off-policy updates recompute log-probs without tips so the policy internalizes memory-guided behavior via reward-weighted distillation, with a novelty-based intrinsic reward for exploration.

Key results

• On ScienceWorld, EMPO² averages 75.9 versus 33.2 for GRPO (the paper reports a 128.6% improvement).
• On WebShop, EMPO² reaches score 88.3 and success 76.9, above GiGPO w/o std (86.2 / 75.2) and GRPO (79.3 / 66.1).
• In OOD adaptation with memory, EMPO² yields an average 136% improvement within 10 steps across three transitions.

Surprising findings

• Offline RL (Retrospex) underperforms non-parametric Reflexion on several ScienceWorld tasks, while online RL never does.
• Off-policy training with tips removed is prone to instability and can diverge to NaN without a low-probability token mask.

Limitations

• Experiments use only Qwen2.5-7B-Instruct; generality across model families and sizes is untested.
• Memory retrieval is a simple cosine-similarity search, which the authors note may limit performance.
• Off-policy updates can be unstable and rely on importance sampling and a masking heuristic.

Experimental setup

Datasets: ScienceWorld WebShop Retrospex offline trajectories

Benchmarks: ScienceWorld WebShop

Metrics: score success rate

Baselines: Reflexion Retrospex GRPO GiGPO

Builds on: GRPO Reflexion Retrospex GiGPO context distillation

Closest papers by meaning

• From Player to Master: Enhancing Test-Time Learning of LLM Agents via Reinforcement Learning over Memory (ICML 2026 · similarity 0.91)
• Agentic Reinforced Policy Optimization (ICLR 2026 · similarity 0.90)
• LLM-Explorer: A Plug-in Reinforcement Learning Policy Exploration Enhancement Driven by Large Language Models (NeurIPS 2025 · similarity 0.90)
• Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions (ICLR 2026 · similarity 0.89)
• Expanding LLM Agent Boundaries with Strategy-Guided Exploration (NeurIPS 2026 · similarity 0.88)
• One-Way Policy Optimization for Self-Evolving LLMs (ICML 2026 · similarity 0.88)