Research map / Multi-agent systems and games / LLM agents

ViLoMem: Agentic Learner with Grow-and-Refine Multimodal Semantic Memory

CVPR 2026 · Multi-agent systems and games › LLM agents · approach: Self-evolving language agent learning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ViLoMem, a training-free dual-stream (visual and logic) error-aware semantic memory that grows and refines schema-based guidelines for MLLM solvers, improving pass@1 on six multimodal benchmarks.

Problem

MLLMs solve each multimodal problem de novo and repeat the same visual and logical mistakes; existing memory agents store single-modality trajectories that suffer from brevity bias and lose visual grounding.

Approach

A dual-stream memory with separate visual guideline memory (paired with source images, retrieved via image-embedding then text-filtering, plus question-aware attention maps) and logic guideline memory (retrieved via LLM problem analysis), updated by similarity-based merge/create in a closed verifier-driven loop.

Key results

• GPT-4.1 with ViLoMem improves MathVision from 47.47 (step) to 53.95 and MathVista from 74.27 to 76.88.
• Qwen3-VL-8B-Instruct with ViLoMem improves MMMU from 65.52 (step) to 69.90 and RealWorldQA from 70.85 to 73.59.
• Memory scaling on WeMath rises from 72.53% (15k tokens) to 74.58% (150k tokens), above direct memory (73.85%).

Surprising findings

• Cross-model memory from other models improved Qwen3-VL-8B more than its self-generated memory (e.g., MMMU 71.26 vs 69.90), suggesting stronger-model memories transfer well.
• Cross-benchmark memory from other tasks hurt some benchmarks (e.g., MMMU 65.14 vs 66.38 baseline), showing domain mismatch interference.

Limitations

• Verifier sometimes attributes all errors to the logic stream when the solver's visual descriptions are weak, causing mixed memory updates.
• Gains on knowledge-intensive benchmarks such as MMMU and MMStar are moderate.
• Cross-domain memory can cause interference for domain-mismatched tasks.

Experimental setup

Datasets: MathGlance MathVista MathVision MathVerse

Benchmarks: MMMU MathVista MathVision HallusionBench MMStar RealWorldQA WeMath

Metrics: pass@1 accuracy

Baselines: GPT-4.1 step-by-step prompting Qwen3-VL-235B-A22B-Instruct step-by-step prompting Qwen3-VL-8B-Instruct step-by-step prompting

Builds on: Dynamic Cheatsheet ACE Reflexion ReAct

Closest papers by meaning

• Navigating Massive Visual Context in Retrieval-Augmented Generation via Multimodal Memory Graph (ICML 2026 · similarity 0.91)
• SMMBench: A Benchmark for Source-Distributed Multimodal Agent Memory (NeurIPS 2026 · similarity 0.91)
• Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memory (ICLR 2026 · similarity 0.91)
• XSkill: Continual Learning from Experience and Skills in Multimodal Agents (ICML 2026 · similarity 0.90)
• MemGen: Weaving Generative Latent Memory for Self-Evolving Agents (ICLR 2026 · similarity 0.90)
• WorldMemArena: Evaluating Multimodal Agent Memory Through Action–World Interaction (NeurIPS 2026 · similarity 0.90)