Research map / Multi-agent systems and games / LLM agents

MedMemoryBench: Benchmarking Agent Memory in Personalized Healthcare

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Software engineering agent benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MedMemoryBench, a human-agent collaboratively built long-horizon medical dialogue benchmark with an evaluate-while-constructing streaming protocol and noise augmentation, used to benchmark 12+ agent memory methods and expose memory saturation.

Problem

Existing agent memory benchmarks focus on general daily conversations and do not capture the precision, safety-critical personalization, longitudinal clinical progression, streaming accumulation, and noise demands of personalized healthcare memory.

Approach

A multi-stage pipeline generates year-long synthetic patient trajectories from 20 chronic-disease profiles with injected trap events, expert-verified QA, and a streaming evaluate-while-constructing protocol with Efficient vs. Mixed (noise-injected) settings to measure memory saturation.

Key results

• Letta achieves the best average accuracy under the Efficient setting (51.21) and Mixed setting (41.55) on MedMemoryBench with GPT-5.1 backbone, while all methods remain far from satisfactory on IG and MCD.
• Under the Mixed (noise-injected) setting, most methods degrade; e.g., MemOS drops from 48.16 to 40.98 average accuracy and A-Mem drops from 44.84 to 29.27.
• LLM-as-judge agreement with human annotators is 87.5% overall (Cohen's kappa 0.72) on 200 sampled responses.

Surprising findings

• Memory saturation: more sessions and injected noise degrade performance for most memory methods, and long-term memory without filtering often hurts overall performance.
• Graph-structured memory methods underperform simple flat or long-context baselines, and a Core Memory design (Letta) outperforms more elaborate retrieval stores.

Limitations

• Patient profiles are synthetic and built from only 20 chronic-disease patients, limiting coverage.
• LLM-as-judge shows weaker agreement on IG (recall 0.58), indicating judge strictness on memory grounding differences.

Experimental setup

Datasets: LoCoMo

Benchmarks: MedMemoryBench LoCoMo

Metrics: accuracy Recall@k F1 score ROUGE-L hallucination rate

Baselines: Long-Context BM25 RAG Embedding RAG GraphRAG HippoRAG-v2 ReMem Mem0 A-Mem MemOS MemRL LightMem Letta

Builds on: LoCoMo Mem0 MemGPT HippoRAG GraphRAG

Closest papers by meaning

• Experience Makes Skillful: Enabling Generalizable Medical Agent Reasoning via Self-Evolving Skill Memory (NeurIPS 2026 · similarity 0.89)
• PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments (NeurIPS 2026 · similarity 0.88)
• MedAgentBoard: Benchmarking Multi-Agent Collaboration with Conventional Methods for Diverse Medical Tasks (NeurIPS 2025 · similarity 0.87)
• MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Making (NeurIPS 2024 · similarity 0.86)
• MedChain: Bridging the Gap Between LLM Agents and Clinical Practice with Interactive Sequence (NeurIPS 2025 · similarity 0.86)
• MMedAgent-RL: Optimizing Multi-Agent Collaboration for Multimodal Medical Reasoning (ICLR 2026 · similarity 0.86)