Research map / Vision-language and multimodal / Multimodal LLMs
CoMemo: LVLMs Need Image Context with Image Memory
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CoMemo, a dual-path LVLM (autoregressive context path plus gated cross-attention memory path) with RoPE-DHR positional encoding that aligns thumbnail-based 2D positions for high-resolution tiles, improving captioning, long-context and multi-image benchmarks over InternVL-2-style baselines at 2B scale.
Problem
LVLMs inherited from LLMs neglect middle visual content as context grows ("lost in the middle") and RoPE over dynamic high-resolution image tokens causes remote decay and loss of 2D structure.
Approach
Combine an autoregressive image context path with a gated cross-attention image memory path (trained in three stages to balance the two), and replace per-token RoPE for high-resolution tiles with RoPE-DHR, which maps tile patches to thumbnail patch position IDs to compress positions and keep 2D layout.
Key results
- CoMemo (2B) reaches 98.6 CIDEr on COCO vs 79.1 for the InternVL-2-style LVLM-S baseline, and 78.8 on NoCaps vs 60.0.
- On MileBench CoMemo scores 54.6 vs 52.1 for LVLM-S (about 2.5% gain) even without DHR, and on MM-NIAH-T 34.6 vs 27.3.
- Training throughput drops from 13.4 samples/s (LVLM-S) to 12.26 samples/s (CoMemo) on 64 A100s; ablation Variant 5 gives overall 60.5 vs 55.4 for the baseline variant.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add RoPE-DHR to the context path without memory (ablation) | CoMemo · Long-generation and long-context tasks | score | Helped |
| Add the memory path without RoPE-DHR (ablation) | CoMemo · Ours benchmark suite | Overall score: 55.4 -> 59.0 | Helped |
| Add RoPE-DHR to the memory-path variant (ablation) | CoMemo · Ours benchmark suite | Overall score: 59.0 -> 60.5 | Helped |
| Use RoPE-DHR without compression instead of compressed RoPE-DHR (ablation) | CoMemo with memory · Ours benchmark suite | Overall score: 59.5 -> 60.5 | No clear effect |
| Add memory and RoPE-DHR at 8B scale (ablation) | CoMemo · Ours benchmark suite | Overall score: 65.4 -> 68.0 | Helped |
| Add memory and RoPE-DHR with 1.2M fine-tuning data (ablation) | CoMemo · Ours benchmark suite | Overall score: 51.1 -> 54.6 | Helped |
| Use compressed RoPE-DHR (ablation) | CoMemo with memory · OCR-related tasks | OCR-related score: 74.8 -> 73.4 | Hurt |
| Use the dual-path architecture instead of LVLM-S | CoMemo · MileBench | score: 52.1 -> 54.6 | Helped |
| Use the dual-path architecture instead of LVLM-S | CoMemo · MMBench, MME, and MMVP | score: 73.1 -> 74.2; 1869 -> 1904; 31.3 -> 36.0 | Helped |
| Use the dual-path architecture instead of LVLM-S | CoMemo · AI2D, ChartQA, and TextVQA | score: 74.3 -> 74.2; 75.6 -> 73.6; 74.2 -> 72.6 | Hurt |
| Use CoMemo instead of LVLM-S | CoMemo · Caption benchmarks | CIDEr: 79.1 -> 98.6; 65.4 -> 78.5; 60.0 -> 78.8 | Helped |
| Use CoMemo instead of LVLM-S | CoMemo · Long-generation benchmarks | score: 62.9 -> 66.9; 28.7 -> 38.7 | Helped |
Surprising findings
- LVLM-X (cross-attention) outperforms the autoregressive LVLM-S on pure captioning, while LVLM-S is stronger on knowledge-intensive tasks.
- Prolonged pretraining causes over-reliance on the cross-attention memory path, which fine-tuning does not fully correct.
Limitations
- Performance on OCR-related tasks (AI2D, ChartQA, TextVQA) is slightly below LVLM-S; RoPE-DHR compression can hurt fine-grained information.
- Results are mostly architecture ablations at 2B scale with limited scale study (only one 8B variant), and the 8B result is not compared to a full suite of baselines.
- Dual-path balance is sensitive to pretraining length and gate freezing, requiring a three-stage schedule.
Experimental setup
Datasets: Laion COYO COCO Wukong-OCR LaionCOCO-OCR GRIT Objects365 All-Seeing VQAv2 GQA ChartQA TextVQA
Benchmarks: COCO Flickr30k NoCaps LLaVABench MMDU BLINK Mantis MMT MM-NIAH MileBench MathVista MathVision
Metrics: CIDEr accuracy score
Baselines: LVLM-X (cross-attention mixin) LVLM-S (InternVL-2 style) InternVL-2 InternVL-2.5 Qwen2-VL mPlug-owl3 LLaMA-3.2-Vision MiniCPM-V-2 GPT-4o GPT-4V
Builds on: LLaVA Flamingo InternVL-2 NVLM-H mPlug-owl3 RoPE
Closest papers by meaning
- POINTS-Long: Adaptive Dual-Mode Visual Reasoning in MLLMs (CVPR 2026 · similarity 0.88)
- Towards General Continuous Memory for Vision-Language Models (NeurIPS 2025 · similarity 0.87)
- Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs (NeurIPS 2024 · similarity 0.87)
- LoMo: Local Modality Substitution for Deeper Vision-Language Fusion (NeurIPS 2026 · similarity 0.87)
- Identifying and Mitigating Position Bias of Multi-image Vision-Language Models (CVPR 2025 · similarity 0.87)
- Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Models (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.