Research map / Multi-agent systems and games / LLM agents

EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspective

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Self-evolving language agent learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces EvoMemBench, an open benchmark that evaluates agent memory along in-episode vs. cross-episode scope and knowledge vs. execution content, and compares 15 memory methods against long-context LLM baselines.

Problem

Existing agent memory benchmarks do not jointly evaluate in-episode and cross-episode memory evolution while separating knowledge-oriented and execution-oriented demands, and some are not open-sourced.

Approach

A unified benchmark built from six reconstructed or selected datasets organized along two axes (memory scope and memory content), with a standardized protocol that reuses source pipelines and swaps in memory modules with utilize/update interfaces.

Key results

• On InEp-Exec at 16K context, ReasoningBank achieves 43.0% overall success vs. 28.5% for DeepSeek-V3.2 backbone (+14.5 points).
• On CrossEp-Know hard split, ACE reaches 13.0% overall vs. 0.0% for DeepSeek-V3.2 without memory.
• On the CrossEp-Know easy split, the memory-free DeepSeek-V3.2 (52.1%) outperforms all general long-term memory methods (43.8-46.8%).

Surprising findings

• Long-context baselines remain highly competitive; Gemini-3-Flash without memory ranks first on InEp-Know.
• Memory can hurt on easy knowledge tasks and at larger context budgets (e.g., several memory methods fall below the no-memory baseline at 128K).

Limitations

• Memory methods are evaluated with a single backbone (DeepSeek-V3.2) for memory-augmented agents, limiting generality across backbones.
• Several datasets are reconstructed with GPT-5-mini assistance and manual verification, and some subsets are small (e.g., 19-46 ALFWorld samples per category).

Experimental setup

Datasets: MemoryAgentBench BFCL-MultiTurn-LongContext BFCL-MultiTurn-Base CL-Bench xbench-DeepSearch WebWalkerQA ALFWorld

Benchmarks: EvoMemBench (InEp-Know, InEp-Exec, CrossEp-Know, CrossEp-Tool, CrossEp-Web, CrossEp-Emb)

Metrics: answer accuracy success rate token usage average rank average steps

Baselines: Gemini-3-Flash GPT-5-mini DeepSeek-V3.2 BM25 Qwen3-Emb-4B GraphRAG MemAgent MemoBrain Mem0 A-MEM MemOS MemoryOS

Builds on: MemoryAgentBench BFCL CL-Bench ALFWorld MemEvolve

Closest papers by meaning

• Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions (ICLR 2026 · similarity 0.95)
• AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications (ICML 2026 · similarity 0.94)
• MemEvolve: Meta-Evolution of Agent Memory Systems (ICML 2026 · similarity 0.93)
• A-Mem: Agentic Memory for LLM Agents (NeurIPS 2025 · similarity 0.93)
• Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks (ICML 2026 · similarity 0.93)
• AdaMEM: Test-Time Adaptive Memory for Language Agents (ICML 2026 · similarity 0.92)