Research map / Vision-language and multimodal / Multimodal LLMs

Navigating Massive Visual Context in Retrieval-Augmented Generation via Multimodal Memory Graph

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes VimRAG, a multimodal RAG agent that models reasoning as a DAG memory graph, allocates visual tokens by graph-based energy, and trains with graph-pruned policy optimization (GGPO) to improve credit assignment.

Problem

Multimodal retrieval-augmented agents rely on linear interaction histories that become token-heavy with visual data, causing state blindness, memory-visual mismatch, and coarse outcome-only rewards in multi-step RAG.

Approach

Represent the agent's reasoning as a dynamic DAG memory graph; score visual memory nodes by topological out-degree, temporal decay, and recursive feedback from children to allocate high-resolution vision tokens; and mask gradients on dead-end steps in positive trajectories and valuable retrievals in negative ones.

Key results

• VimRAG reaches 50.1 overall accuracy on Qwen3-VL-8B-Instruct, versus 43.6 for Mem1 and 37.6 for vanilla RAG.
• On Qwen3-VL-4B-Instruct, VimRAG reaches 45.2 overall versus 40.6 for Mem1.
• Ablation: graph memory raises accuracy from 43.6 to 47.1; adding multimodal memory and energy-based allocation brings it to 50.1.

Surprising findings

• Retaining raw selective visual tokens outperforms textual captioning for memory (58.2% vs 52.8% on image tasks in the pilot study), while storing all raw visual tokens performs poorly.

Limitations

• Performance depends on the capability of the base model.
• Multi-turn interaction may be unsuitable for high real-time applications.
• Retriever accuracy still needs improvement.

Experimental setup

Datasets: HotpotQA SQuAD WebQA SlideVQA MMLongBench LVBench WikiHowQA SyntheticQA HowTo100M

Benchmarks: HotpotQA SQuAD WebQA SlideVQA MMLongBench LVBench WikiHowQA SyntheticQA XVBench

Metrics: binary accuracy (0/1) judged by model

Baselines: Vanilla RAG ReAct VideoRAG UniversalRAG MemAgent Mem1

Builds on: ReAct GSPO Mem1 MemAgent VideoRAG

Closest papers by meaning

• ViLoMem: Agentic Learner with Grow-and-Refine Multimodal Semantic Memory (CVPR 2026 · similarity 0.91)
• ARK: A Dual-Axis Multimodal Retrieval Benchmark along Reasoning and Knowledge (NeurIPS 2026 · similarity 0.91)
• SMMBench: A Benchmark for Source-Distributed Multimodal Agent Memory (NeurIPS 2026 · similarity 0.90)
• Fix Before Search: Benchmarking Agentic Visual Query Pre-processing in Multimodal Retrieval-augmented Generation (ICML 2026 · similarity 0.89)
• Memory is Reconstructed, Not Retrieved: Graph Memory for LLM Agents (ICML 2026 · similarity 0.89)
• ThinkMorph: Emergent Properties in Multimodal Interleaved Chain-of-Thought Reasoning (ICLR 2026 · similarity 0.89)