Research map / Vision-language and multimodal / Multimodal LLMs

Reading Images Like Texts: Sequential Image Understanding in Vision-Language Models

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Analyzes how VLM visual encoders recognize objects layer by layer via logit lens token maps and how 2D RoPE encodes spatial relations, then proposes run-length token compression with a distilled visual decoder and a RoPE scaling trick for spatial reasoning.

Problem

Understanding how VLMs process images as 1D token sequences, how they recognize objects across discontinuous patches, and how positional encodings (absolute and 2D RoPE) represent 2D spatial relations; also the hallucination and efficiency issues this raises.

Approach

Applies logit lens to ViT layers to read images as text token maps, revealing a two-stage attribute-recognition-then-semantic-disambiguation process; theoretically derives the geometry of 2D RoPE spatial encoding; uses these to build an instruction-agnostic run-length token compression with a distilled visual decoder and a RoPE frequency scaling g(i)=1+alpha(2i/d)^p.

Key results

• Run-length compression with punctuation removal on LLaVA-1.5-7B reduces visual tokens by 58.35% with GQA 60.14 vs 60.50 original and POPE 85.66 vs 85.96.
• RoPE scaling on Qwen2-VL-2B raises What's Up B from 53.16 to 58.25 training-free, and from 61.52 to 63.48 when combined with SFT.
• Logit-lens analysis shows LLaVA-1.5-7B POPE accuracy near random until about ViT layer 11-12, then improving steadily.

Surprising findings

• Removing punctuation-decoded visual tokens preserves most benchmark accuracy for LLaVA-1.5-7B (e.g., POPE 85.66 vs 85.96) while cutting tokens by up to 58%.
• Qwen2-VL-7B with SFT plus RoPE scaling scores slightly lower on GQA-spatial 2 (94.5) than SFT alone (95.42).

Limitations

• Logit lens applies the final connector to intermediate ViT layers, which was not trained for them, so the layer-wise interpretation may be imperfect.
• Spatial analysis only covers four basic relations (left, right, front, behind) with a simplified two-object setup.
• Compression reduction rate and accuracy trade-off is weaker than instruction-aware methods on compression ratio and task performance; RoPE scaling is only a localized remedy.

Experimental setup

Datasets: GQA TextVQA POPE What's Up VSR COCO-spatial GQA-spatial VQA v2.0 ScienceQA MMBench

Benchmarks: VQA v2.0 GQA ScienceQA TextVQA MMBench-EN POPE What's Up A/B VSR COCO-spatial GQA-spatial

Metrics: accuracy reduction rate (%) precision recall

Baselines: ToME LLaVA-1.5-7B Qwen2.5-VL-7B Qwen2-VL-2B Qwen2-VL-7B

Builds on: logit lens RoPE LLaVA-1.5 Qwen2.5-VL knowledge distillation

Closest papers by meaning

• Towards Interpreting Visual Information Processing in Vision-Language Models (ICLR 2025 · similarity 0.92)
• Aligning What Vision-Language Models See and Perceive with Adaptive Information Flow (CVPR 2026 · similarity 0.91)
• What's in the Image? A Deep-Dive into the Vision of Vision Language Models (CVPR 2025 · similarity 0.91)
• Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.91)
• To Sink or Not to Sink: Visual Information Pathways in Large Vision-Language Models (ICLR 2026 · similarity 0.90)
• The Narrow Gate: Localized Image-Text Communication in Native Multimodal Models (NeurIPS 2025 · similarity 0.90)