Research map / Vision-language and multimodal / Multimodal LLMs

VisMem: Latent Vision Memory Unlocks Potential of Vision-Language Models

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes VisMem, which gives VLMs short-term (visual) and long-term (semantic) latent vision memory tokens invoked during autoregressive decoding, trained in two GRPO-based stages, reporting an 11.0% average gain over Qwen2.5-VL-7B.

Problem

VLMs lose grounding in visual evidence during prolonged autoregressive generation (a "visual processing bottleneck"), and existing interventions (direct RL training, image-level tools, token-level selection, language-only latent reasoning) are costly, forgetful, or lack visual memory.

Approach

Inspired by a cognitive memory theory distinguishing visually dominant short-term memory from semantically dominant long-term memory, it adds special invocation/end tokens to the VLM vocabulary; on invocation a lightweight query builder turns the hidden states into a query, and one of two LoRA-based memory formers produces latent memory tokens that are inserted back into the generation stream.

Key results

• VisMem on Qwen2.5-VL-7B averages 68.2 on understanding, reasoning and generation-type benchmarks, an 11.0% average improvement over the vanilla model (vs. 62.5 for Vision-R1's average under the table's averaging).
• On MMVet with four-stage continual learning, VisMem retains 72.1% at stage 3 versus 68.4% for DeepEyes and 67.0% for Mirage.
• Across nine base models (3B to 38B), VisMem improves every listed benchmark, e.g. Qwen2.5-VL-7B MV-Math rises from 18.9 to 41.4.

Surprising findings

• Full invocation at 100% lowers performance on MuirBench (56.0) and MV-Math (17.5) relative to random 75% invocation, indicating excessive memory invocation hurts.

Limitations

• Latency overhead of 8.2-43.8% over the vanilla model remains non-trivial.
• Relies on benchmark training splits and additional reasoning data for training, and gains depend on the chosen memory lengths and invocation rates.

Experimental setup

Datasets: Visual CoT Mulberry MMVet BLINK MuirBench MMMU LogicVista MathVista MV-Math HallBench MultiTrust MMVU

Benchmarks: MMStar MMVet MMT-Bench BLINK MuirBench MMMU LogicVista MathVista MV-Math HallBench MultiTrust MMVU

Metrics: accuracy average score inference time inference speed

Baselines: SFT Visual-RFT VLM-R1 Vision-R1 PAPO Sketchpad GRIT PixelReasoner DeepEyes OpenThinkImg Scaffold ICoT

Builds on: GRPO Visual-RFT VLM-R1 Coconut Mirage

Closest papers by meaning

• Dual-Latent Memory Routing for Vision-Language Reasoning (ICML 2026 · similarity 0.91)
• Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training (ICLR 2026 · similarity 0.91)
• Testing the Limits of Fine-Tuning for Improving Visual Cognition in Vision Language Models (ICML 2025 · similarity 0.90)
• Memory-Space Visual Prompting for Efficient Vision-Language Fine-Tuning (ICML 2024 · similarity 0.90)
• Towards General Continuous Memory for Vision-Language Models (NeurIPS 2025 · similarity 0.89)
• Elevating Visual Perception in Multimodal LLMs with Visual Embedding Distillation (NeurIPS 2025 · similarity 0.89)