Research map / Multi-agent systems and games / LLM agents
WorldMemArena: Evaluating Multimodal Agent Memory Through Action–World Interaction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes WorldMemArena, a 461-task multimodal multi-session benchmark that evaluates agent memory through a four-stage lifecycle (write, maintain, retrieve, use) across long-context, manually designed, and harness-based memory agents.
Problem
Existing memory benchmarks test static dialogue recall, report only end-of-task accuracy, and convert visual observations to captions, so they cannot localize where agent memory fails or evaluate self-managed memory in interactive multimodal settings.
Approach
Formulates multimodal agent memory as an Action-World Interaction Loop with an observable four-stage lifecycle, and annotates multi-session tasks with gold memory points, state updates, distractors, and evidence chains so each stage can be scored separately.
Key results
- Qwen3-VL-Embedding-8B reaches 86.22% memory recall and 98.15% memory correctness, yet its QA-correct rate is only 51.86%, showing good storage does not guarantee good answers.
- Across long-context agents, DeepSeek V4 achieves 69.13% QA-correct, while Claude Haiku 4.5 reaches only 36.71%.
- Harness agent Codex-GPT 5.4-nano attains 53.62% QA-correct and 32.56 F1, outperforming most manually designed memory systems but at high latency cost (up to ~786 s per task for SimpleMem).
Surprising findings
- High memory recall and correctness coexist with low downstream QA accuracy, as storage quality does not predict answer performance.
- Most memory systems rely on append-only updates and remain weak on updating stale facts and rejecting interference.
Limitations
- Harness-based agents are evaluated end-to-end only because their internal memory process is hard to decompose.
- Memory and QA metrics depend on an LLM judge (GPT-5.4-mini), which may introduce judge bias.
Experimental setup
Datasets: EmbodiedBench VisualAgentBench Agent-Arena GUI task collection in-house long-horizon dialogue collection
Benchmarks: WorldMemArena
Metrics: memory recall memory correctness memory hallucination update handling interference rejection QA correct QA hallucination QA omission F1 BLEU-1
Baselines: Qwen3-VL-Embedding-8B UniversalRAG A-Mem MemGPT SimpleMem Omni-SimpleMem M2A ViLoMem MIRIX AUGUSTUS OpenClaw Codex
Builds on: LoCoMo LongMemEval MemGPT Mem0 UniversalRAG
Closest papers by meaning
- SMMBench: A Benchmark for Source-Distributed Multimodal Agent Memory (NeurIPS 2026 · similarity 0.93)
- AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications (ICML 2026 · similarity 0.92)
- Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks (ICML 2026 · similarity 0.92)
- Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions (ICLR 2026 · similarity 0.91)
- EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspective (NeurIPS 2026 · similarity 0.91)
- G-Memory: Tracing Hierarchical Memory for Multi-Agent Systems (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.