Research map / Vision-language and multimodal / Multimodal LLMs

The Narrow Gate: Localized Image-Text Communication in Native Multimodal Models

NeurIPS 2025 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Compares native multimodal VLMs (Chameleon, Emu3) with non-native VLMs (LLaVA, Pixtral, Janus, VILA-U) and finds native models route image-to-text information through a single end-of-image token ("narrow gate"), which can be ablated or patched to steer predictions.

Problem

How visual information is transferred from image tokens to text tokens inside vision-language models, and whether native multimodal models that generate both images and text communicate differently from text-output VLMs.

Approach

Native multimodal VLMs separate image and text representations in the residual stream and concentrate text-to-image attention and semantic content on one post-image [EOI] token, whereas non-native models distribute cross-modal communication across many image tokens; the [EOI] token can be causally ablated or patched to steer image semantics.

Key results

• Ablating the [EOI] token to text in Chameleon-7B drops VQAv2 accuracy from 0.51 to 0.25 and Flickr30k from 0.34 to 0.04, while blocking all text-to-image attention in LLaVA drives VQAv2 to 0.00.
• Patching [EOI] representations from a target class into base-class inputs in Chameleon-7B/34B reaches similarity 0.86 and changes the predicted class in about 90% of cases (Emu3: ~75%).
• In Chameleon-7B, the [EOI] token receives 40-50% of text-on-image attention between layers 2 and 6, and its neighborhood overlap with ImageNet labels exceeds 0.4.

Measured effects reported in the paper

Surprising findings

• Blocking text attention to the [EOI] token in LLaVA leaves performance unchanged, while blocking all image attention collapses it to near zero.
• Attention to the [EOI] token is high in native models but its semantic content is not necessarily the largest in non-native models, where internal image tokens carry the signal.

Limitations

• Analysis limited to the image-to-text direction and a subset of VLM architectures.
• Does not evaluate text-to-image communication, native VLMs with continuous encoders, or diffusion-based decoders.
• Only small-scale evaluation subsets (2,000 examples per dataset) are used.

Experimental setup

Datasets: VQAv2 Flickr30k MS-COCO ImageNet LLaVA-Instruct-150K

Benchmarks: VQAv2 Flickr30k MS-COCO ImageNet

Metrics: VQA accuracy CIDEr neighborhood overlap homogeneity score cosine similarity distribution similarity (Jaccard-style) class-switch success rate

Baselines: LLaVA-onevision-7B Pixtral-12B Janus-1.3B VILA-U-7B

Builds on: attention knockout activation patching advanced density peaks clustering memory tokens

Closest papers by meaning

• Towards Interpreting Visual Information Processing in Vision-Language Models (ICLR 2025 · similarity 0.92)
• Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations (NeurIPS 2025 · similarity 0.91)
• LMFusion: Adapting Pretrained Language Models for Multimodal Generation (NeurIPS 2025 · similarity 0.91)
• UVU: Improving Multimodal Understanding via Vision-Language Unified Autoregressive Paradigm (CVPR 2026 · similarity 0.91)
• MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizer (ICLR 2026 · similarity 0.91)
• VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks (ICLR 2025 · similarity 0.91)