Research map / Vision-language and multimodal / Multimodal LLMs

Aligning What Vision-Language Models See and Perceive with Adaptive Information Flow

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a training-free, test-time method (AIF) that uses token-dynamics entropy to identify irrelevant visual tokens and masks text-to-visual attention via a modulated causal mask, improving perception in LLaVA-1.5 and Qwen2.5-VL.

Problem

VLMs often attend to the correct image region but still answer incorrectly, because text tokens spread attention over irrelevant visual tokens during decoding, creating a misalignment between seeing and perceiving.

Approach

Measure each visual token's importance via entropy over its layer-wise token dynamics (max text-attention per layer), then block attention from text tokens to high-entropy (irrelevant) visual tokens by modulating the causal mask at inference time, with the mask ratio chosen by how far the post-mask attention entropy moves from the original distribution. Requires only one extra decoding step and no

Key results

• Qwen2.5-VL-7B AIF improves V* from 78.5 to 84.8 and MMStar from 63.9 to 70.9 without retraining.
• LLaVA-1.5-7B AIF raises RealWorldQA from 55.6 to 60.5 and POPE from 85.4 to 88.7.
• Qwen2.5-VL-7B AIF improves average RefCOCO-family grounding accuracy by 2.3 points (86.6 to 88.9 overall).

Surprising findings

• Masking high-attention visual tokens (10% ratio) causes over 50% relative drop on TextVQA, while masking low-attention tokens barely changes performance.
• Future-aware causal masks (vis2vis and vis2text variants) degrade performance rather than help.

Limitations

• May struggle with lengthy or indirect text prompts, where it is hard to capture important regions.
• Sensitivity to mask ratio varies across datasets; OCR tasks are more sensitive to masking.

Experimental setup

Datasets: RealWorldQA TextVQA CountBench RefCOCO POPE COCO

Benchmarks: V* RealWorldQA MMStar TextVQA SeedBench2-Plus CountBench RefCOCO RefCOCO+ RefCOCOg POPE

Metrics: accuracy recall of object tokens

Baselines: ViCrop CCA Grounding-DINO-L Localization Heads Shikra-7B InternVL2-8B InternVL-2.5-8B GPT-4o Claude-3.5 Sonnet

Builds on: ViCrop concentric causal attention future-aware causal mask VLMEvalKit

Closest papers by meaning

• Towards Interpreting Visual Information Processing in Vision-Language Models (ICLR 2025 · similarity 0.92)
• Rethinking Visual Token Reduction in LVLMs Under Cross-Modal Misalignment (AAAI 2026 · similarity 0.92)
• Reading Images Like Texts: Sequential Image Understanding in Vision-Language Models (ICLR 2026 · similarity 0.91)
• Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMs (ICLR 2026 · similarity 0.91)
• What's in the Image? A Deep-Dive into the Vision of Vision Language Models (CVPR 2025 · similarity 0.91)
• A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMs (CVPR 2025 · similarity 0.90)