Research map / Multi-agent systems and games / LLM agents

Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MemoryArena, a multi-session benchmark gym of human-crafted interdependent agentic tasks (web shopping, group travel planning, progressive web search, math/physics formal reasoning) that evaluates agent memory inside memory-agent-environment loops, finding that long-context, memory-system, and RAG agents all achieve low success rates.

Problem

Existing agent memory evaluations either test recall over static long-context QA or evaluate single-session agentic actions, so they do not measure whether memory built through interaction guides later actions in interdependent multi-session tasks.

Approach

Treats memorization and action as coupled by evaluating agents across sequential sessions where subtasks causally depend on earlier interactions, with persistent memory updated after each session and retrieved for later ones, measured by end-to-end success rate and progress scores.

Key results

• Across memory paradigms with GPT-5.1-mini as task agent, average success rate is low (e.g., long-context avg 0.16 over all tasks, memory avg 0.13, RAG best ~0.20 with Text-Embedding-3-Small), and Group Travel Planning SR is near zero for all methods.
• External memory and RAG do not consistently beat long-context history; retrieval helps mainly in Progressive Web Search and Formal Reasoning.
• Success rate decays as subtask depth increases across all methods; with oracle-like memory injected into long-context agents, gains are small (e.g., +0.05 on Travel and Search).

Surprising findings

• Agents with near-saturated scores on LoCoMo perform poorly in the agentic multi-session setting.
• Long-context agents, with all information preserved, still perform poorly, suggesting failure to utilize rather than failure to observe is a main bottleneck.

Limitations

• Limited number of tasks in math and physics (40 and 20 groups) due to expensive expert annotation.
• Only a few task agents evaluated fully; main results use GPT-5-mini/5.1-mini, with a single additional stronger-agent check.
• Group Travel Planning yields near-zero success for all methods, limiting discriminability.

Experimental setup

Datasets: WebShop BrowseComp-Plus TravelPlanner

Benchmarks: MemoryArena

Metrics: success rate progress score soft progress score latency

Baselines: Letta Mem0 Mem0-g Mirix ReasoningBank BM25 Text-Embedding-3-Small MemoRAG GraphRAG

Builds on: WebShop BrowseComp-Plus TravelPlanner LoCoMo

Closest papers by meaning

• EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspective (NeurIPS 2026 · similarity 0.93)
• Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions (ICLR 2026 · similarity 0.92)
• AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications (ICML 2026 · similarity 0.92)
• WorldMemArena: Evaluating Multimodal Agent Memory Through Action–World Interaction (NeurIPS 2026 · similarity 0.92)
• G-Memory: Tracing Hierarchical Memory for Multi-Agent Systems (NeurIPS 2025 · similarity 0.91)
• MemGen: Weaving Generative Latent Memory for Self-Evolving Agents (ICLR 2026 · similarity 0.91)