Research map / Vision-language and multimodal / Multimodal LLMs
Towards General Continuous Memory for Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CoMEM, which fine-tunes a VLM (via LoRA plus a shared-parameter Q-Former) into a continuous memory encoder that compresses retrieved multimodal knowledge into 8 embeddings prepended to a frozen VLM, trained on 15.6K self-synthesized samples.
Problem
VLMs struggle with knowledge-intensive multimodal and multilingual reasoning, and concatenating retrieved image-text knowledge as discrete tokens inflates context length and can degrade performance.
Approach
Use the VLM itself as a continuous memory encoder: its hidden states over retrieved knowledge items are compressed by a lightweight shared-parameter Q-Former into 8 dense embeddings per item, prepended to a frozen VLM, with only LoRA and Q-Former trained on self-synthesized data.
Key results
- CoMEM on Qwen2VL reaches 38.7 average across six English benchmarks vs 27.8 for base Qwen2-VL and 31.1 for Qwen2-VL+RAG.
- CoMEM on Qwen2.5-VL averages 35.4 across English benchmarks vs 30.8 base and 27.6 with RAG.
- On multilingual InfoSeek, CoMEM raises overall accuracy on Qwen2.5-VL from 16.7 to 24.2 and on Qwen2-VL from 14.6 to 23.4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add CoMEM memory to Qwen2-VL | Qwen2-VL · OK-VQA | accuracy: 36.3 -> 57.7 | Helped |
| add CoMEM memory to Qwen2-VL | Qwen2-VL · A-OKVQA | accuracy: 41.8 -> 60.6 | Helped |
| add CoMEM memory to Qwen2.5-VL | Qwen2.5-VL · OK-VQA | accuracy: 35.0 -> 47.6 | Helped |
| add CoMEM memory to Qwen2.5-VL | Qwen2.5-VL · A-OKVQA | accuracy: 39.8 -> 55.0 | Helped |
| add CoMEM memory to Qwen2.5-VL | Qwen2.5-VL · MRAG-Bench | accuracy: 42.0 -> 38.1 | Hurt |
| add CoMEM memory to Qwen2-VL | Qwen2-VL · MRAG-Bench | accuracy: 39.3 -> 35.1 | Hurt |
| add CoMEM memory | Qwen2.5-Instruct-VL · Multilingual InfoSeek Overall | accuracy: 16.7 -> 24.2 | Helped |
| add CoMEM memory | Qwen2.5-Instruct-VL · CVQA Overall | accuracy: 67.45 -> 68.53 | Helped |
| add CoMEM memory | Qwen2-Instruct-VL · Multilingual InfoSeek Overall | accuracy: 14.6 -> 23.4 | Helped |
| add CoMEM memory | Qwen2-Instruct-VL · CVQA Overall | accuracy: 66.14 -> 67.57 | Helped |
| increase training data to 2x or 4x (ablation) | CoMEM on Qwen2.5-VL · InfoSeek All | accuracy: 30.7 -> 30.9; 30.7 -> 31.3 | No clear effect |
| reduce training data to 0.5x or 0.25x (ablation) | CoMEM on Qwen2.5-VL · InfoSeek All | accuracy: 30.7 -> 25.4; 30.7 -> 17.6 | Hurt |
Surprising findings
- Vanilla RAG often degrades performance below the no-retrieval VLM baseline (e.g., Qwen2.5-VL on InfoSeek Query 22.5 to 17.7).
- Even without any training, reusing the VLM's own cached hidden states as memory outperforms RAG and FastV on several benchmarks.
Limitations
- Evaluation benchmarks are mostly static and synthetic; dynamic or noisy real-world inputs are not tested.
- Only evaluated in a single-model setting; multi-agent or cross-model memory sharing is unexplored.
- Trainable parameter count (~200M) is non-trivial, and no statistical significance or error bars are reported.
Experimental setup
Datasets: InfoSeek Encyclopedic-VQA OK-VQA WIT
Benchmarks: InfoSeek OVEN MRAG-Bench OK-VQA A-OKVQA ViQuAE CVQA Multilingual InfoSeek
Metrics: accuracy exact match
Baselines: LLaVA-v1.5 LLaVA-v1.6 LLaMA3 InternLM-XComposer2.5 mPLUG-Owl3 Qwen2-VL Qwen2.5-VL Wiki-LLaVA RORA-VLM EchoSight ReflectiVA FastV
Builds on: retrieval-augmented generation FastV BLIP-2 Q-Former LoRA RoRA-VLM
Closest papers by meaning
- VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks (ICLR 2025 · similarity 0.91)
- Dual-Latent Memory Routing for Vision-Language Reasoning (ICML 2026 · similarity 0.91)
- LION: Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge (CVPR 2024 · similarity 0.91)
- Too Late to Recall: Explaining the Two-Hop Problem in Multimodal Knowledge Retrieval (NeurIPS 2025 · similarity 0.91)
- Vision‑Language‑Vision Auto‑Encoder: Scalable Knowledge Distillation from Diffusion Models (NeurIPS 2025 · similarity 0.91)
- Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.