Research map / Vision-language and multimodal / Multimodal LLMs

When Language Overrules: Revealing Text Dominance in Multimodal Large Language Models

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Training-free token reduction for MLLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Attention-level study showing text dominance across image, video, audio, time-series and graph MLLMs using the MDI and AEI metrics, attributing it to token redundancy, fusion design and task formulation, and mitigating it with [CLS]-guided visual token pruning (FasterVLM) on LLaVA-1.5-7B.

Problem

MLLMs over-attend to text tokens while underusing non-text inputs; it is unclear whether this text dominance extends beyond vision-language to audio, video, time-series and graph modalities and what causes it.

Approach

Proposes two attention-based indices, the Modality Dominance Index (MDI, per-token text attention vs. non-text attention) and the Attention Efficiency Index (AEI, attention share over token share), applies them across five modalities, and shows that reducing non-text tokens via [CLS]-guided pruning rebalances attention.

Key results

• VideoLLaMA3-7B reaches a late-layer MDI of 157.53 on MMBench-Video, and Qwen2.5-VL-7B's image MDI rises from 2.26 (early) to 33.10 (late) on MMMU-Pro.
• Replicating non-text tokens tenfold raises ChatTS-14B's late-layer MDI from 3.52 to 16.25 and Qwen2-Audio's middle-layer MDI from 3.24 to 10.10.
• FasterVLM at 90% token reduction on LLaVA-1.5-7B lowers late-layer MDI from 17.37 to 1.84 (abstract cites 10.23 to 0.86 at the middle layers in the compressed setting).

Surprising findings

• GraphGPT-7B shows MDI below 1 (graph-favoring) at baseline, and only shifts to text dominance when graph tokens are replicated tenfold.
• Simpler LLaVA-1.5 shows higher AEI than the more integrated Qwen2.5-VL despite lower MDI, indicating a control-versus-efficiency trade-off.

Limitations

• The token compression method is FasterVLM applied only to LLaVA-1.5-7B on one benchmark; the proposed mitigation is not evaluated on other modalities or task accuracy.
• Attention-based MDI/AEI may not causally reflect model reliance on modalities; the causal claims rest on correlational analysis.
• Inconsistent numbers across the abstract, text and tables (e.g., MDI 10.23 vs 17.37, 0.86 vs 1.84) and the graph replication results are reported with limited detail.

Experimental setup

Datasets: MMMU-Pro MMBench-Video IEMOCAP TimeSeries-Reasoning GraphGPT-Eval-Instruction

Benchmarks: MMMU-Pro MMBench-Video IEMOCAP TimeSeries-Reasoning GraphGPT-Eval-Instruction

Metrics: Modality Dominance Index (MDI) Attention Efficiency Index (AEI)

Baselines: FasterVLM LLaVA-1.5-7B

Builds on: FasterVLM Modality Importance Score Data Remixing MBPO

Closest papers by meaning

• Rethinking Visual Token Reduction in LVLMs Under Cross-Modal Misalignment (AAAI 2026 · similarity 0.90)
• Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings (NeurIPS 2025 · similarity 0.90)
• The Alignment Illusion in Multimodal Large Language Models (NeurIPS 2026 · similarity 0.90)
• Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs (ICML 2026 · similarity 0.89)
• What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Models (CVPR 2026 · similarity 0.89)
• TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs (NeurIPS 2026 · similarity 0.89)