Research map / Vision-language and multimodal / Multimodal LLMs

Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes modality-mutual attention (MMA), which unlocks the causal mask so image tokens can attend to text query tokens during SFT, improving vision-language alignment without extra parameters.

Problem

Vision-language misalignment (object hallucination, weak vision-centric reasoning) in MLLMs, partly caused by causal attention that prevents image tokens placed before text from seeing the text query.

Approach

Modify the LLM attention mask so image tokens can attend to question text tokens (modality-mutual attention) during SFT, instead of training with both image-text and text-image orders (dual-order training), which doubles cost; the change adds no parameters and no compute.

Key results

• MMA improves over the (I&T)PT+(I&T)SFT baseline on all three LLM backbones; on Phi-3.5-Mini, MME-P rises from 1226.3 to 1363.7 and MMMU from 31.1 to 39.4.
• On LLaMA-3.2-3B, MMA reaches 1258.2 MME-P and 37.1 MMMU, outperforming CCA and MA by an average of about 6.2% across 12 benchmarks (per abstract).
• AKI-4B outperforms 7B-scale models with comparable training data on most benchmarks, e.g., POPE 88.1 and RealWorldQA 62.9 at 3B scale.

Surprising findings

• Dual-order training (DOT) underperforms CCA and MA on several benchmarks despite doubling training time, and some ablated variants outperform full DOT on specific tasks.

Limitations

• MMA only applies during SFT; applying it in pretraining causes information leakage through image tokens to future text tokens.
• Experiments use small models (3B-8B), short training schedules and limited SFT data, and do not support interleaved inputs.
• Results on knowledge-intensive benchmarks such as MMMU remain behind larger MLLMs.

Experimental setup

Datasets: BLIP3-kale BLIP3-OCR-200M VQAv2 GQA VSR OCRVQA A-OKVQA ScienceQA RefCOCO RefCOCOg RefCOCO+ VisualGenome

Benchmarks: MME MMBench SEED-Bench LLaVA-Bench MMMU MathVista POPE MM-Vet RealWorldQA CV-Bench

Metrics: accuracy MME perception score MME cognition score LLM-judged score

Baselines: Concentric Causal Attention (CCA) Mixed-Attention (MA) Dual-order training (DOT) (I&T)PT + (I&T)SFT causal baseline BLIP-3-4B Qwen2-VL-2B MM1.5-3B MiniCPM-V2-3B Phi-3-Vision-4B LLaVA-1.5-7B

Builds on: Concentric Causal Attention Mixed-Attention Perceiver Resampler OpenFlamingo

Closest papers by meaning

• TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs (NeurIPS 2026 · similarity 0.92)
• Where Does Vision Meet Language? Understanding and Refining Visual Fusion in MLLMs via Contrastive Attention (CVPR 2026 · similarity 0.92)
• Wings: Learning Multimodal LLMs without Text-only Forgetting (NeurIPS 2024 · similarity 0.92)
• Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs (NeurIPS 2024 · similarity 0.92)
• Unveiling Visual Perception in Language Models: An Attention Head Analysis Approach (CVPR 2025 · similarity 0.92)
• Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception (CVPR 2024 · similarity 0.91)