Research map / Vision-language and multimodal / Multimodal LLMs

Rethinking Causal Mask Attention for Vision-Language Inference

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Bias-aware multi-image robustness · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Investigates relaxing causal masks for visual tokens in VLM prefill, proposes future-aware masks (full, v2v, v2t) and a lightweight kernel-pooling merge of future attention into prefix sink tokens, evaluated on MILEBench with LLaVA-7B/13B.

Problem

Strict causal masking inherited from text-only LLMs prevents visual tokens in VLMs from attending to future tokens, possibly limiting contextual semantic capture during prefill.

Approach

Relax causal masks for visual queries to preview future visual and/or textual tokens (three variants), and a lightweight variant that kernel-pools future attention scores and merges them into the initial prefix token(s) so the final attention stays strictly causal and decoding latency is reduced.

Key results

• Relaxed masks (M^f, M^v2v, M^v2t) improve several temporal and text-rich tasks over the causal baseline on LLaVA-7b and 13b, but degrade some retrieval/text tasks.
• Merging pooled future attention into the prefix reduces decoding latency from 83.18 to 26.53 ms/token for M^f on the reported setup while keeping performance near the unmerged variants.
• Merging into the first token alone already yields strong results, per the prefix-ratio analysis.

Surprising findings

• Relaxing causal constraints on visual tokens can unexpectedly improve performance even though the model was trained causally.
• Merging future attention into a single prefix token sometimes matches or beats direct future access on some tasks.

Limitations

• Mask strategy is selected manually per task rather than learned or adapted automatically.
• Relaxed masks degrade performance on text-dominant or retrieval tasks (S-5, IR), so gains are not uniform.
• Mutual-information properties in the appendix rest on stated assumptions (causally isotropic representations) and are not empirically validated.

Experimental setup

Datasets: MILEBench

Benchmarks: MILEBench

Metrics: accuracy ROUGE-L decoding latency (ms/token)

Baselines: standard causal mask

Builds on: LLaVA FlashAttention MILEBench

Closest papers by meaning

• Mitigating Mask Prior Drift and Positional Attention Collapse in Large Diffusion Vision-Language Models (ICML 2026 · similarity 0.91)
• Causal Tracing of Object Representations in Large Vision Language Models: Mechanistic Interpretability and Hallucination Mitigation (AAAI 2026 · similarity 0.89)
• Large Vision-Language Models Get Lost in Attention (ICML 2026 · similarity 0.89)
• Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training (ICLR 2026 · similarity 0.89)
• The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models Via Visual Information Steering (ICML 2025 · similarity 0.89)
• Dismantling Pathological Shortcuts: A Causal Framework for Faithful LVLM Decoding (ICML 2026 · similarity 0.89)