Research map / Multi-agent systems and games / LLM agents

MEME: Multi-Entity & Evolving Memory Evaluation

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Agent data synthesis and behavior studies · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MEME, a DAG-based benchmark with six tasks (incl. Cascade, Absence, Deletion) for multi-entity and evolving memory in LLM agents, and finds that six memory systems fail dependency reasoning, with only file-based MD-flat plus Opus 4.7 partially closing the gap at ~70x cost.

Problem

Existing memory benchmarks evaluate only independent single-entity updates and do not test whether dependent facts are correctly updated, marked uncertain, or removed after an upstream change in persistent LLM agent memory.

Approach

Organize memory evaluation along entity scope (single vs multi-entity) and temporal dynamics (static vs evolving), and generate controlled episodes from a DAG knowledge graph with explicit conditional rules so cascade and absence gold answers are verifiable by construction.

Key results

• All six memory systems fail dependency reasoning: average Cascade 0.03 and Absence 0.01 under the default gpt-4.1-mini configuration, with the best overall system MD-flat at 0.42.
• MD-flat with Claude Opus 4.7 as internal LLM reaches Cascade 0.32 and Absence 0.59 on a 20-episode subset, at ~70x the baseline cost ($4.54 vs $0.065 per episode).
• Prompt optimization (DSPy SIMBA), top-k up to 40, no-filler condition, and answering-LLM swap to Sonnet 4 do not close the Cascade gap.

Surprising findings

• Deeper retrieval, prompt optimization, and a stronger answering LLM fail to improve Cascade, pointing to a failure in the memory layer rather than the answerer.
• In-context gpt-4.1-mini (0.36 overall) outperforms five of six memory systems on accuracy, though it costs more per query.

Limitations

• Only two hand-crafted knowledge graphs (Personal Life, Software Project), and dialogues are LLM-generated rather than from real users.
• Most ablations run on 10-40 episode subsets and cover only a subset of systems due to compute cost.
• Explicit conditional phrasing for dependency rules is a best-case framing; implicit-conditional variants were not tested.

Experimental setup

Datasets: LongMemEval-S ShareGPT coding English subset

Benchmarks: MEME

Metrics: accuracy cost per episode

Baselines: BM25 text-embedding-3-small Mem0 Graphiti Karpathy Wiki MD-flat in-context gpt-4.1-mini in-context Sonnet 4.6

Builds on: LoCoMo LongMemEval MemoryAgentBench DSPy SIMBA Karpathy LLM knowledge base

Closest papers by meaning

• Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions (ICLR 2026 · similarity 0.89)
• A-Mem: Agentic Memory for LLM Agents (NeurIPS 2025 · similarity 0.88)
• SimpleMem: Efficient Lifelong Memory for LLM Agents (ICML 2026 · similarity 0.88)
• MEMAUDIT: An Exact Package-Oracle Evaluation Protocol for Budgeted Long-Term LLM Memory Writing (NeurIPS 2026 · similarity 0.88)
• EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspective (NeurIPS 2026 · similarity 0.88)
• MemForest: Efficient Agent Memory Management via EventTree Partitioning and Progressive Merging (NeurIPS 2026 · similarity 0.87)