Research map / Multi-agent systems and games / LLM agents

MemArena: An Ego-Centric Benchmark for On-Device Agentic Personal Memory Assistants at Scale

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Self-evolving language agent learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MemArena, an ego-centric, activity-dense conversational memory benchmark generated by the MASim multi-agent simulator (50 agents, 15 days), and evaluates five open-weight readers with Vanilla, BM25-RAG, Oracle, Memobase, and MemSearch memory backends across six recall, reasoning, and trustworthiness dimensions plus edge latency.

Problem

Existing memory benchmarks do not test personal on-device memory assistants jointly on ego-centric perspective, globally coherent multi-session worlds, and human-scale activity density, nor permission-aware disclosure from memory.

Approach

A day-batched multi-agent simulator (MASim) generates person-person and person-agent dialogs from a shared social world, with provenance metadata, so ground-truth evaluation instances are built alongside the data and each user sees only an ego-centric projection; permission-aware access is scored with an F1 over privacy and utility.

Key results

• At Qwen3-8B, switching Memobase to MemSearch raises Recall by +22.1 pp and Reasoning by +21.2 pp, while scaling the reader to Qwen3-32B-AWQ gives at most +3.5/+4.4 pp.
• No deployable backend-reader cell exceeds F1_PU = 19 on permission-aware access, while Oracle readers disclose the protected fact on 75-99% of DENY items.
• Memory search adds a fixed 87/8/51 ms (BM25-RAG/Memobase/MemSearch), which is 54% of TTFT only for Qwen3-0.6B with BM25 and 3.3% for Qwen3-32B-AWQ.

Surprising findings

• Oracle readers larger than Qwen3-0.6B disclose protected facts more often, indicating better extraction but no permission judgment.
• Deployable backends' low leakage stems from missing utility (not surfacing facts) rather than policy-based gating; policy refusals are only 0.6% of DENY responses.

Limitations

• Corpus is synthetic and English-only, so it underrepresents conversational noise, code-switching, and cultural privacy norms.
• Results rely on LLM-judge scoring and LLM-generated factual QA labels; some backend differences flip sign under a second judge.
• Single 15-day world with a single privacy-norm setting; cross-world variance is not characterized.

Experimental setup

Datasets: MemArena-L (synthetic, generated by MASim)

Benchmarks: MemArena-L

Metrics: exact-match accuracy accuracy F1_PU (privacy-utility harmonic mean) TTFT (ms)

Baselines: Vanilla context BM25-RAG Oracle retrieval Memobase MemSearch

Builds on: LoCoMo LongMemEval BM25 Memobase contextual integrity

Closest papers by meaning

• AMemGym: Interactive Memory Benchmarking for Assistants in Long-Horizon Conversations (ICLR 2026 · similarity 0.89)
• LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory (ICLR 2025 · similarity 0.88)
• Mem-PAL: Towards Memory-based Personalized Dialogue Assistants for Long-term User-Agent Interaction (AAAI 2026 · similarity 0.87)
• Personalize-then-Store: Benchmarking and Learning Personalized Memory for Long-Horizon Agents (NeurIPS 2026 · similarity 0.86)
• MemSim: A Bayesian Simulator for Evaluating Memory of LLM-based Personal Assistants (NeurIPS 2025 · similarity 0.86)
• MemForest: Efficient Agent Memory Management via EventTree Partitioning and Progressive Merging (NeurIPS 2026 · similarity 0.85)