Research map / Multi-agent systems and games / LLM agents

SMMBench: A Benchmark for Source-Distributed Multimodal Agent Memory

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Self-evolving language agent learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces SMMBench, a 1,877-sample benchmark where multimodal evidence is scattered across independent sources (chats, profiles, tables, images, documents), and evaluates memory-style and RAG-style agents on cross-source reasoning, conflict resolution, preference reasoning, and function calling.

Problem

Existing multimodal memory benchmarks evaluate reasoning within pre-assembled or coherent contexts and do not test whether agents can compose evidence distributed across independently originated multimodal sources.

Approach

Builds QA instances whose answer-critical evidence spans at least two independent sources, inserts evidence into simulated multi-source long conversations with LLM-based source-aware insertion, and measures memory and RAG systems on five task types.

Key results

• Best overall score among benchmarked baselines is HMRAG at 0.4933, versus 0.7473 for the Golden Evidence Baseline.
• Function Call is the hardest task: best F.C. score is 0.1111 (HMRAG), and the Golden Evidence Baseline reaches only 0.2778.
• Native RAG reaches Recall@100 of 0.7903 but Precision@100 of only 0.0207, showing retrieval noise grows with budget.

Surprising findings

• MIRIX improves from 0.2496 to 0.3161 overall with native multimodal access, while MemGPT drops from 0.3708 to 0.2713 when moving to native MM.
• Even the golden evidence baseline reaches only 0.2778 on function calling, indicating difficulty beyond retrieval.

Limitations

• Conversational sources are synthetically generated by LLM agents, which may not match real user chat histories.
• Evaluation relies on multiple-choice formats for most tasks, and the Function Call task uses exact match, which may be strict.

Experimental setup

Datasets: ChartQA_Pro SlideVQA MMDocRAG MMCV MLLMKC MMKE MMPB LifeSim M3Bench

Benchmarks: SMMBench

Metrics: multiple-choice accuracy exact match Recall@K Precision@K

Baselines: Short-Term Mem. Reflexion Mem. Gen. Agent Mem. Self Controlled Mem. MIRIX MemGPT MemVerse Mem0 OmniSimpleMem Native RAG HMRAG UniversalRAG

Builds on: LoCoMo Mem0 MemGPT MIRIX VRAG

Closest papers by meaning

• WorldMemArena: Evaluating Multimodal Agent Memory Through Action–World Interaction (NeurIPS 2026 · similarity 0.93)
• ViLoMem: Agentic Learner with Grow-and-Refine Multimodal Semantic Memory (CVPR 2026 · similarity 0.91)
• Navigating Massive Visual Context in Retrieval-Augmented Generation via Multimodal Memory Graph (ICML 2026 · similarity 0.90)
• Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memory (ICLR 2026 · similarity 0.90)
• EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspective (NeurIPS 2026 · similarity 0.90)
• AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications (ICML 2026 · similarity 0.89)