Research map / Vision-language and multimodal / Multimodal LLMs

Where Does Vision Meet Language? Understanding and Refining Visual Fusion in MLLMs via Contrastive Attention

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Training-free token reduction for MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Layer-wise visual masking analysis of seven MLLMs reveals shallow-to-middle fusion layers and a late "review" phenomenon, and a training-free contrastive attention method (final vs. Hellinger-selected pre-integrated layer) segments and re-inputs image regions to improve VQA benchmarks.

Problem

How visual and textual information is progressively fused across layers inside MLLMs is poorly understood, and existing training-free visual refinement methods rely on manually chosen layers and are affected by persistent high-attention noise.

Approach

Identify visual fusion layers via layer-wise image-token masking and attention Hellinger distance, then contrast the attention of an automatically selected pre-integrated layer against the final layer to localize text-guided regions, crop them, and re-input the image.

Key results

• On LLaVA-Next, the proposed method reaches 68.65 average accuracy versus 66.22 base and 67.73 for OPERA across the seven benchmarks in the table.
• On Qwen2.5-VL, average accuracy improves from 75.15 (base) to 77.66, the best among compared training-free methods (ViCrop 77.29).
• On LLaVA-1.5, average accuracy rises from 56.34 to 59.16, outperforming OPERA (58.11) and ViCrop (57.60).

Surprising findings

• Certain models (LLaVA-1.5, LLaVA-Next, InstructBLIP, VIP-LLaVA) show a late-stage 'review' layer where masking visual tokens again causes an accuracy drop.
• Unconstrained pre-integrated layer selection concentrates heavily on layer 2 (over 86% of DocVQA samples).

Limitations

• Pre-integrated layer selection strategies differ in effectiveness, and the best (fusion-layer constrained) requires the empirically identified fusion layers.
• Method requires a second inference pass with re-input of segmented images, adding compute overhead.
• Gains on some benchmarks are small or negative (e.g., DocVQA and MMBench for certain models).

Experimental setup

Datasets: VQAv2 GQA TextVQA OKVQA VizWiz DocVQA

Benchmarks: GQA VQAv2 OKVQA VizWiz TextVQA DocVQA MMBench

Metrics: accuracy average accuracy inference time

Baselines: CD DoLA OPERA ViCrop

Builds on: ViCrop contrastive decoding DoLA OPERA

Closest papers by meaning

• Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs (ICML 2026 · similarity 0.92)
• Look-Back: Implicit Visual Re-focusing in MLLM Reasoning (AAAI 2026 · similarity 0.92)
• Unveiling Visual Perception in Language Models: An Attention Head Analysis Approach (CVPR 2025 · similarity 0.92)
• Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Models (CVPR 2026 · similarity 0.91)
• Lifting the Veil on Visual Information Flow in MLLMs: Unlocking Pathways to Faster Inference (CVPR 2025 · similarity 0.91)
• Understanding Language Prior of LVLMs by Contrasting Chain-of-Embedding (ICLR 2026 · similarity 0.91)