Research map / Vision-language and multimodal / Multimodal LLMs

Where MLLMs Attend and What They Rely On: Explaining Autoregressive Token Generation

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Training-free token reduction for MLLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes EAGLE, a black-box, greedy-search attribution framework that explains which image superpixel regions drive autoregressive token generation in MLLMs and quantifies language-prior vs. visual-evidence reliance.

Problem

Explaining which visual regions an MLLM's autoregressive token generation depends on, and how much tokens rely on language priors versus perceptual evidence, is poorly understood; existing gradient and activation methods are memory-heavy, model-specific, or limited to single tokens.

Approach

A submodular-inspired objective summing an insight score (probability with selected regions kept) and a necessity score (probability drop when they are removed), optimized by greedy ordering of SLICO superpixels; an influence score tracks how token probability evolves as regions are added to measure visual vs. prior reliance.

Key results

• On MS COCO captioning with LLaVA-1.5 7B, EAGLE reaches 0.5970 sentence-level insertion vs. 0.5298 for LLaVA-CAM and uses 16.07 GB vs. 37.25 GB for LLaVA-CAM.
• On Qwen2.5-VL 7B word-level MS COCO, EAGLE achieves Point Game box 0.7785 vs. 0.5369 for TAM.
• On RePOPE with Qwen2.5-VL 7B, EAGLE reduces AMCR to 0.1442 from a best baseline of 0.5357 (iGOS++) and reaches CSR@10% of 73.94%.

Surprising findings

• TAM performs well on Qwen2.5-VL and InternVL3.5 but poorly on LLaVA-1.5, indicating model-dependence of activation-based attribution.
• EAGLE is roughly ten times slower than LLaVA-CAM and iGOS++ in attribution time (e.g., 436 s vs. 22-63 s).

Limitations

• Iterative greedy subset selection limits scalability; attribution time is substantially higher than baselines.
• Focuses on hallucination explanation and partial mitigation rather than prevention.
• Objective is not strictly submodular for MLLMs, so greedy search is a heuristic.

Experimental setup

Datasets: MS COCO MMVP RePOPE

Benchmarks: MS COCO MMVP RePOPE

Metrics: insertion AUC deletion AUC average highest score Point Game (box) Point Game (mask) AMCR CSR@10% GPU memory time

Baselines: LLaVA-CAM iGOS++ (w/ GNC) TAM

Builds on: VPS TAM I-GOS++ SLICO

Closest papers by meaning

• What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Models (CVPR 2026 · similarity 0.90)
• Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings (NeurIPS 2025 · similarity 0.90)
• Rethinking Visual Token Reduction in LVLMs Under Cross-Modal Misalignment (AAAI 2026 · similarity 0.90)
• POINTS-Long: Adaptive Dual-Mode Visual Reasoning in MLLMs (CVPR 2026 · similarity 0.90)
• Fill the GAP: A Granular Alignment Paradigm for Visual Reasoning in Multimodal Large Language Models (NeurIPS 2026 · similarity 0.89)
• UVU: Improving Multimodal Understanding via Vision-Language Unified Autoregressive Paradigm (CVPR 2026 · similarity 0.89)