Research map / Multi-agent systems and games / LLM agents

Experience-Evolving Multi-Turn Tool-Use Agent with Hybrid Episodic–Procedural Memory

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Robust and efficient agent execution · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes H-EPM, a tool graph with episodic state summaries on edges and efficiency-weighted procedural transitions, used to guide multi-turn tool selection at inference and to bias GRPO rollouts for RL on tau-bench, tau2-bench and ToolSandbox.

Problem

Multi-turn tool-use agents face shifting decision contexts and partial observability; existing reuse is either trajectory-level (too context-specific) or tool-level (ignores state), and long-horizon RL explores ineffectively.

Approach

A hybrid memory graph whose nodes are tools and whose edges carry both efficiency-weighted transition statistics (procedural) and compact LLM-generated state summaries (episodic); the agent adaptively chooses between similarity-based episodic recall and weight-based procedural suggestions, and the same memory biases RL rollouts with a random skip rate.

Key results

• On tau2-bench with GPT-4.1, H-EPM reaches 0.512 vs 0.358 base (+43.0%), and with Qwen3-8B 0.309 vs 0.201 (+53.7%).
• RL with Qwen3-4B-Instruct-2507 on tau2-bench improves from 0.158 base and 0.178 GRPO to 0.223 with H-EPM (skip rate 1.0), +41.1% relative over base.
• On tau-bench RL, H-EPM (skip 0.9) reaches 0.558 vs 0.510 for GRPO and 0.435 base; SFT on H-EPM trajectories drops to 0.373.

Surprising findings

• SFT on successful H-EPM trajectories degrades performance (e.g., tau-bench 0.435 base to 0.373), attributed to overfitting.
• Masking guidance tokens in the loss outperforms discarding them, with the paper arguing a rollout-update mismatch is more harmful than train-test mismatch.

Limitations

• Stronger reasoning models adhere less to external guidance, limiting gains relative to instruction-tuned models.
• Explicit summarization decisions incur higher inference cost than implicit invocation.
• ToolNet was re-implemented by the authors since the original code is not open-sourced.

Experimental setup

Datasets: tau-bench (Retail/Telecom) tau2-bench (Retail) ToolSandbox

Benchmarks: tau-bench tau2-bench ToolSandbox ACEBench

Metrics: task success rate process accuracy end-to-end accuracy

Baselines: Base model ReasoningBank ToolNet Retrieval Top-3 GRPO SFT AgentEvolver

Builds on: GRPO ReasoningBank ToolNet AgentEvolver GNN4Plan

Closest papers by meaning

• Skill-Pro: Learning Reusable Skills from Experience via Non-Parametric PPO for LLM Agents (ICML 2026 · similarity 0.89)
• MemEvolve: Meta-Evolution of Agent Memory Systems (ICML 2026 · similarity 0.88)
• Procedural Memory Distillation: Online Reflection for Self-Improving Language Models (NeurIPS 2026 · similarity 0.88)
• Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks (ICML 2026 · similarity 0.87)
• MemGen: Weaving Generative Latent Memory for Self-Evolving Agents (ICLR 2026 · similarity 0.87)
• MemRL: Self-Evolving Agents via Runtime Reinforcement Learning on Episodic Memory (NeurIPS 2026 · similarity 0.87)