Research map / Multi-agent systems and games / LLM agents

Harness the Memory: A Holistic Evaluation of Memory Substrates in Memory Agents

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Controlled harness evaluation of 11 memory substrate methods (seven families) across three backbones and four benchmarks (LoCoMo, LME-S, MAB, ALFWorld, BigCodeBench-Hard), with 26 metrics, finding no single substrate dominates and that retrieval breadth and history length reverse the best choice across regimes.

Problem

Existing memory evaluations for LLM agents are concentrated on dialogue QA benchmarks, report little efficiency data, and use mostly GPT backbones, so it is unclear which memory substrate should be used under which operating regime.

Approach

Isolate the memory substrate as the sole experimental variable under a shared harness (backbone, prompts, auxiliary LLM fixed) and compare dense/sparse indices, text, structural, hierarchical, refinement, weight and activation substrates on both user-centric QA and agent-centric decision tasks, with retrieval-depth sweeps, attention probes, and context-length scaling.

Key results

• On LoCoMo and LME-S, M5 (dual-level graph) achieves the best P4 across backbones (e.g., 0.648 on Qwen3-8B LoCoMo) but at 10-100x the latency of flat baselines.
• On ALFWorld with Qwen3-32B-AWQ, M7 distilled strategies reach 32.1% TSR versus 22.4% for NoMem, while flat retrievers M1 and M2 fall below it at higher k.
• On LoCoMo, P4 rises monotonically with retrieval depth k, whereas on ALFWorld task success falls as k grows from 1 to 5.

Surprising findings

• Retrieval breadth helps long-context QA but hurts sequential embodied decision-making, with attention shifting away from the task observation toward retrieved memory.
• Flat BM25 retrieval (M2) gives cheap, competitive results and lifts BigCodeBench-Hard Pass@1 above NoMem on every backbone, while the heavier graph method M5 is dominated on cost-performance for agentic tasks.

Limitations

• Auxiliary LLM fixed to gpt-4o-mini, so results may not transfer to other aux models
• Mem0, Zep and MemGPT evaluated only in an auxiliary-cost analysis rather than the main comparison
• M8 uses a zero-shot LLM instead of the original PPO controller and M11 deviates from the EpiCache implementation

Experimental setup

Datasets: LoCoMo LongMemEval-S MemoryAgentBench ALFWorld BigCodeBench-Hard AgentGym successes

Benchmarks: LoCoMo LongMemEval-S MemoryAgentBench ALFWorld BigCodeBench-Hard

Metrics: LLM-judge score per-query latency task success rate Pass@1 steps to goal SubEM token usage cost

Baselines: NoMem Dense Vector Sparse Vector Full Context Mem0 Zep MemGPT

Builds on: A-Mem LightRAG RAPTOR ReadAgent ReasoningBank MemSkill EpiCache LoRA

Closest papers by meaning

• Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions (ICLR 2026 · similarity 0.93)
• A-Mem: Agentic Memory for LLM Agents (NeurIPS 2025 · similarity 0.91)
• AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications (ICML 2026 · similarity 0.91)
• PlugMem: A Task-Agnostic Plugin Memory Module for LLM Agents (ICML 2026 · similarity 0.91)
• EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspective (NeurIPS 2026 · similarity 0.90)
• AdaMEM: Test-Time Adaptive Memory for Language Agents (ICML 2026 · similarity 0.90)