Research map / Multi-agent systems and games / LLM agents

Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: Self-evolving language agent learning · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MemoryAgentBench, a multi-turn incremental benchmark for memory agents covering accurate retrieval, test-time learning, long-range understanding, and selective forgetting, and evaluates long-context, RAG, and agentic memory agents, finding that all methods struggle with selective forgetting.

Problem

Existing benchmarks evaluate LLM agent reasoning and long-context reading in static single-block settings, while memory agents that incrementally absorb and update information in multi-turn interactions lack a comprehensive evaluation covering retrieval, learning, understanding, and forgetting.

Approach

Reconstruct existing long-context datasets into sequential multi-turn chunks fed incrementally with memorization instructions, add two new datasets (EventQA for accurate retrieval, FactConsolidation for selective forgetting), and map the datasets to four memory competencies grounded in cognitive science to evaluate long-context, RAG, and agentic memory agents under a unified protocol.

Key results

• Long-context agents (GPT-5-mini) achieve the best TTL and LRU scores, while RAG methods like BM25 and HippoRAG-v2 lead on accurate retrieval (e.g., HippoRAG-v2 65.1 AR average).
• All evaluated methods fail on multi-hop FactConsolidation, reaching at most 28% accuracy (GPT-5-mini 28.0 on FC-MH).
• Upgrading MIRIX's backbone from GPT-4o-mini to GPT-4.1-mini raises its overall average across four datasets from 15.9 to 25.6.

Surprising findings

• RAG-based agents generally underperform their GPT-4o-mini backbone on test-time learning tasks.
• o4-mini reaches 80.0 on 6K FactCon-MH but drops to 14.0 at 32K, showing long-context reasoning limits even for reasoning models.

Limitations

• Due to budget constraints, only a subset of representative memory agents was evaluated.
• Some datasets (EventQA, FactConsolidation) are synthetic or derived from existing benchmarks, which may limit realism.
• Evaluation relies on GPT-4o as a judge for open-ended answers.

Experimental setup

Datasets: SQuAD-style document QA (SH-Doc QA, MH-Doc QA) LongMemEval EventQA (from infinity-Bench books) BANKING77 CLINC150 NLU TREC Coarse TREC Fine Redial movie recommendation infinity-Bench En.Sum Detective QA MQuAKE

Benchmarks: MemoryAgentBench LongMemEval (S*) EventQA FactConsolidation-SH FactConsolidation-MH infinity-Bench En.Sum Detective QA Redial movie recommendation

Metrics: accuracy SubEM Recall@5 F1-score LLM-judged accuracy

Baselines: GPT-4o GPT-4o-mini Claude-3.7-Sonnet BM25 Contriever HippoRAG-v2 MemGPT MIRIX

Builds on: LongMemEval LOCOMO MQuAKE ∞-Bench MemGPT

Closest papers by meaning

• EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspective (NeurIPS 2026 · similarity 0.95)
• A-Mem: Agentic Memory for LLM Agents (NeurIPS 2025 · similarity 0.95)
• AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications (ICML 2026 · similarity 0.94)
• Harness the Memory: A Holistic Evaluation of Memory Substrates in Memory Agents (NeurIPS 2026 · similarity 0.93)
• MemGen: Weaving Generative Latent Memory for Self-Evolving Agents (ICLR 2026 · similarity 0.92)
• Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks (ICML 2026 · similarity 0.92)