Research map / Multi-agent systems and games / LLM agents

Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memory

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces M3-Agent, a multimodal agent with entity-centric long-term memory built online from video/audio streams and RL-trained multi-turn memory retrieval, plus the M3-Bench long-video QA benchmark.

Problem

Multimodal agents need to continuously perceive streams, store consistent long-term memory of entities and world knowledge, and reason over it; existing long-video QA benchmarks and single-turn RAG do not evaluate or support this well.

Approach

Two parallel processes: a memorization model that builds episodic and semantic memory clip-by-clip into an entity-centric multimodal graph using face/voice IDs, and a control model that iteratively searches this memory and is trained with DAPO reinforcement learning using GPT-4o-judged rewards.

Key results

• M3-Agent reaches 30.7% accuracy on M3-Bench-robot, 48.9% on M3-Bench-web, and 61.8% on VideoMME-long, outperforming the strongest baseline Gemini-GPT4o-Hybrid (24.0, 41.2, 56.5).
• Removing semantic memory drops accuracy by 17.1%, 19.2%, and 13.1% on the three benchmarks.
• RL (DAPO) training improves control accuracy by 10.0%, 8.0%, and 9.3% over the prompted 32B control model.

Surprising findings

• Prompted GPT-4o and Gemini-based agents underperform specialized online video methods such as MA-LMM on some robot-video metrics, and a 7B memory model with SFT beats Gemini-1.5-Pro prompted memory.

Limitations

• Fine-grained detail retention is hard, with memory risking cognitive overload.
• Spatial reasoning is weak since verbal memory is less effective than visual memory for spatial information.
• M3-Bench-robot uses human actors wearing head cameras instead of real robots.

Experimental setup

Datasets: M3-Bench-robot M3-Bench-web VideoMME-long in-house training videos

Benchmarks: M3-Bench-robot M3-Bench-web VideoMME-long

Metrics: accuracy

Baselines: Gemini-1.5-Pro GPT-4o Qwen2.5-Omni-7b Qwen2.5-VL-7b MovieChat MA-LMM Flash-VStream Gemini-Agent Gemini-GPT4o-Hybrid

Builds on: DAPO Socratic Models Qwen2.5-Omni Qwen3 retrieval-augmented generation

Closest papers by meaning

• ViLoMem: Agentic Learner with Grow-and-Refine Multimodal Semantic Memory (CVPR 2026 · similarity 0.91)
• Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions (ICLR 2026 · similarity 0.90)
• WorldMemArena: Evaluating Multimodal Agent Memory Through Action–World Interaction (NeurIPS 2026 · similarity 0.90)
• Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Tasks (NeurIPS 2024 · similarity 0.90)
• SMMBench: A Benchmark for Source-Distributed Multimodal Agent Memory (NeurIPS 2026 · similarity 0.90)
• AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications (ICML 2026 · similarity 0.90)