Research map / Vision-language and multimodal / Multimodal LLMs
Lifting the Veil on Visual Information Flow in MLLMs: Unlocking Pathways to Faster Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes visual information flow in MLLMs via saliency and attention-disruption experiments, finding a shallow-to-deep shift, and proposes HiMAP, a training-free hierarchical image-token pruning method that cuts FLOPs by about 65-75% while preserving accuracy.
Problem
MLLMs spend substantial compute on image tokens, yet how visual information flows through the layers, and whether image tokens are needed at every depth, is unclear.
Approach
Shows visual information flows from image tokens to instruction tokens in shallow layers and is aggregated among image tokens in deeper layers, then prunes image tokens at two depths using a criterion matched to each phase (attention to instruction tokens shallow, attention received from image tokens deep).
Key results
- HiMAP on LLaVA-v1.5-7B keeps VQAv2 at 78.6 (baseline 78.3) at 24% of FLOPs (0.73 vs 2.98 TFLOPs) with K1=2, R1=50%, K2=8, R2=75%.
- With K2=15, R2=75%, HiMAP on LLaVA-v1.5-7B reaches NoCaps CIDEr 78.7 and Flickr30k 51.3, vs baseline 78.8 and 50.9, while FastV gets 78.6 and 50.6 at 54% FLOPs.
- On A800 GPU, HiMAP cuts LLaVA-v1.5-7B latency per example from 0.197s to 0.116s, versus 0.144s for FastV.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Shallow-layer pruning module ranking image tokens by visual-textual attention (img2txt) and deeper-layer module by intra-visual attention (img2img), combined (ablation) | HiMAP on LLaVA-v1.5-7B · ScienceQA / A-OKVQA | Accuracy: ScienceQA 68.3 vs 62.4 for img2img shallow; A-OKVQA 77.2 vs 76.2 for img2txt deep | Helped |
| Shallow-layer pruning ranks image tokens by intra-visual attention instead of visual-textual attention (ablation) | shallow-layer pruning module, LLaVA-v1.5-7B · ScienceQA | Accuracy: 62.4 vs 68.3 | Hurt |
| Deeper-layer pruning ranks by visual-textual attention instead of intra-visual attention (ablation) | deeper-layer pruning module, LLaVA-v1.5-7B · A-OKVQA | Accuracy: 76.2 vs 77.2 | Hurt |
| Adding the shallow-layer pruning module alone (K=2, R=50%) to LLaVA-v1.5-7B (ablation) | LLaVA-v1.5-7B · ScienceQA / A-OKVQA | FLOPs ratio and accuracy: 54% FLOPs; ScienceQA 68.3 vs 67.9 | Helped |
| Adding the deeper-layer pruning module (K2=8, R2=75%) alone to LLaVA-v1.5-7B (ablation) | LLaVA-v1.5-7B · ScienceQA / A-OKVQA | FLOPs ratio and accuracy: 34% FLOPs; ScienceQA 68.1 vs 67.9 | Helped |
| Aggressive deeper pruning (K2=8, R2=75%) vs conservative (K2=15) on captioning (ablation) | HiMAP on LLaVA-v1.5-7B · Nocaps / Flickr30k | CIDEr: Nocaps 76.1 at K2=8,R2=75% vs 78.7 at K2=15,R2=75% | Hurt |
| HiMAP (K1=2,R1=50%,K2=8,R2=75%) vs FastV on short-answer and multiple-choice QA | LLaVA-v1.5-7B · VQAv2, A-OKVQA, ScienceQA | Accuracy: VQAv2 78.6 vs 78.1; A-OKVQA 77.2 vs 77.0 at similar or lower FLOPs | Helped |
| FastV applied to LLaVA-v1.5-13B (K=3, R=50%) | LLaVA-v1.5-13B · A-OKVQA | Accuracy: 81.3 vs baseline 82 | Hurt |
| HiMAP vs FastV on image captioning and natural QA (LLaVA-7B) | LLaVA-v1.5-7B · Nocaps / Flickr30k / LLaVA-Bench / MM-Vet | CIDEr / score: Nocaps 78.7 vs FastV 78.6; LLaVA-Bench 66.5 vs FastV 62.4 and baseline 65.7 | Helped |
| HiMAP vs FastV on real-world inference latency (ScienceQA, A800 GPU) | LLaVA-v1.5-7B · ScienceQA latency | Latency/Example: 0.116s vs FastV 0.144s vs baseline 0.197s | Helped |
| Pruning image tokens in shallow layers by visual-textual importance without intra-visual flow — shallow prune choice on intra-visual flow lowers accuracy | LLaVA-v1.5-7B · Perturbation analysis (Sci-VQA, A-OKVQA) | Score consistency | No clear effect |
Surprising findings
- Image tokens contribute only about 0.03% of the saliency importance of instruction tokens despite making up most of the input.
- Pruning image tokens sometimes slightly improves accuracy over the unpruned baseline (e.g., VQAv2 and MME on LLaVA-7B).
Limitations
- Analysis limited to LLaVA-v1.5 family models in the main text, with pruning ratios and layer choices tuned per task.
- Reported speedups and FLOPs reduction are theoretical estimates; real-time gains measured on a single GPU only.
Experimental setup
Datasets: A-OKVQA ScienceQA
Benchmarks: VQAv2 TextVQA MME POPE ScienceQA A-OKVQA NoCaps Flickr30k LLaVA-Bench MM-Vet ChartQA DocQA
Metrics: accuracy CIDEr TFLOPs FLOPs ratio latency GPU memory label consistency score consistency
Baselines: FastV Baseline (unpruned MLLM)
Builds on: FastV saliency analysis attention sink
Closest papers by meaning
- Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference (AAAI 2025 · similarity 0.92)
- A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMs (CVPR 2025 · similarity 0.92)
- ST3: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimming (AAAI 2025 · similarity 0.91)
- TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Model (CVPR 2025 · similarity 0.91)
- Rethinking Visual Token Reduction in LVLMs Under Cross-Modal Misalignment (AAAI 2026 · similarity 0.91)
- Where Does Vision Meet Language? Understanding and Refining Visual Fusion in MLLMs via Contrastive Attention (CVPR 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.