Research map / Large language models / Agents, code and math
MemSim: A Bayesian Simulator for Evaluating Memory of LLM-based Personal Assistants
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MemSim, a Bayesian-network-based simulator that generates user profiles, messages and verified QAs for evaluating memory of LLM personal assistants, releasing the MemDaily dataset and a memory-mechanism benchmark.
Problem
Objective, automatic evaluation of how well LLM-based personal assistants memorize and use factual information from user messages is lacking, because LLM-generated QAs are unreliable (hallucination), lack diversity, and do not scale.
Approach
A Bayesian Relation Network (causal DAG over entity attributes) is sampled via ancestral sampling to produce diverse hierarchical user profiles; shared structured hints then causally drive both user messages and QAs, so the LLM only rewrites facts rather than inventing them, keeping ground truths reliable.
Key results
- On MemDaily-vanilla, FullMem reaches 0.976 accuracy on simple QAs versus 0.508 for NonMem, while OracleMem reaches 0.966, and FullMem scores only 0.320 on aggregative QAs.
- Human verification of about 20% of MemDaily trajectories gives average textual-answer accuracy of 99.8% and retrieval-target accuracy of 99.8%.
- MemSim's user profiles achieve SWI-A 3.050 versus 0.943 for JointPL and 0.347 for IndePL, with R-Human 4.91 versus 3.02 for JointPL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BRNet to generate hierarchical user profiles | MemSim user-profile generation · User profiles | R-Human: 3.02±1.14 -> 4.91±0.30 | Helped |
| Use BRNet-based profiles rather than profile-generation baselines | MemSim user-profile generation · User profiles | SWI-A: 0.943 -> 3.050 | Helped |
| Apply MemSim's constrained, hint-grounded message generation | User-message generation · User messages | SWIP: 5.868 -> 6.125 | Helped |
| Generate messages and QAs causally from shared factual hints | MemSim QA construction · MemDaily average | Textual answer accuracy: null | Helped |
| Use shared-hint QA construction instead of direct LLM answer generation | MemDaily QA construction · Constructed questions and answers | Ground-truth reliability | Helped |
| Add full profile-attribute constraints without forcing their generation | User-message generation · User messages | Informativeness: 4.01±1.18 -> 4.37±0.98 | Helped |
| Use FullMem rather than Retrieved Memory | GLM-4-9B memory benchmark · MemDaily-vanilla Simp. | Accuracy: 0.898±0.048 -> 0.976±0.022 | Helped |
| Use FullMem rather than Retrieved Memory | GLM-4-9B memory benchmark · MemDaily-100 Simp. | Accuracy: 0.892±0.034 -> 0.962±0.017 | Helped |
| Use embedding retrieval rather than LLM retrieval | Target-message retrieval · MemDaily-100 Comp. | Recall@5: 0.683±0.036 -> 0.778±0.048 | Helped |
| Use LLM retrieval rather than embedding retrieval | Target-message retrieval · MemDaily-vanilla Comp. | Recall@5: 0.845±0.022 -> 0.947±0.018 | Helped |
| Use FullMem rather than NonMem | GLM-4-9B memory benchmark · MemDaily-100 Simp. | Response time: 0.119±0.001 -> 1.632±0.097 | Hurt |
| Use Retrieved Memory with FAISS indexing rather than memory methods that do not build indexes | GLM-4-9B memory benchmark · MemDaily-vanilla Simp. | Adaptation time: < 0.001 -> 0.222±0.009 | Hurt |
Surprising findings
- FullMem unexpectedly outperforms OracleMem on simple questions, suggesting LLMs may perform better with medium-length memory prompts.
- NoisyMem shows higher accuracy than NonMem on MemDaily-vanilla but lower accuracy on MemDaily-100.
Limitations
- Evaluates only factual information, not higher-level abstract information such as hidden hobbies or preferences.
- Does not cover dialogue-form interactions, which are harder to keep reliable.
- Aggregative QAs remain hard for all memory mechanisms, and reliability depends on LLM rewriting which can introduce information deviation.
Experimental setup
Datasets: MemDaily
Benchmarks: MemDaily-vanilla MemDaily-10 MemDaily-50 MemDaily-100 MemDaily-200
Metrics: accuracy Recall@5 response time adaptation time R-Human R-GPT SWI F-Human N-Human I-Human
Baselines: FullMem RetrMem ReceMem NonMem NoisyMem OracleMem JointPL SeqPL IndePL ZeroCons PartCons SoftCons
Builds on: MemoryBank Kahn's algorithm FAISS
Closest papers by meaning
- AMemGym: Interactive Memory Benchmarking for Assistants in Long-Horizon Conversations (ICLR 2026 · similarity 0.88)
- LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory (ICLR 2025 · similarity 0.87)
- Mem-PAL: Towards Memory-based Personalized Dialogue Assistants for Long-term User-Agent Interaction (AAAI 2026 · similarity 0.86)
- MemArena: An Ego-Centric Benchmark for On-Device Agentic Personal Memory Assistants at Scale (NeurIPS 2026 · similarity 0.86)
- MEME: Multi-Entity & Evolving Memory Evaluation (NeurIPS 2026 · similarity 0.86)
- How Does Personalized Memory Shape LLM Behavior? Benchmarking Rational Preference Utilization in Personalized Assistants (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.