Research map / Vision-language and multimodal / Multimodal LLMs

MMInference: Accelerating Pre-filling for Long-Context Visual Language Models via Modality-Aware Permutation Sparse Attention

ICML 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MMInference, a permutation-based dynamic sparse attention method that accelerates pre-filling for long-context vision-language models by exploiting grid-shaped attention heads and modality boundaries, without model modification or fine-tuning.

Problem

Quadratic attention cost during pre-filling makes long-context vision language models slow to first token when processing long multi-modal (video, image, text) inputs.

Approach

Identifies a grid-shaped sparse attention pattern in VLM heads and modality-boundary patterns (Q-boundary, 2D-boundary); uses row/column and modality-wise permutations so that sparse indices become contiguous and can be computed with dense FlashAttention-style kernels, with an offline modality-aware search selecting per-head patterns.

Key results

• MMInference matches full attention on video understanding benchmarks (e.g., Llava-Video-7B avg 57.6 vs 57.6 full attention) at 47.3% FLOPs, about half of MInference's FLOPs.
• Achieves up to 8.3x end-to-end pre-filling speedup over FlashAttention-2 at 1M tokens on a single A100, and 1.7x over MInference.
• Kernel-level grid pattern speedup of 12x; Grid attention latency is 358ms at 1M tokens.

Measured effects reported in the paper

Surprising findings

• VLMs show lower attention sparsity than text-only LLMs (1.79% vs 5.78% of weights for 95% recall in the reported comparison), with bottom layers especially dense.
• Sparse indices reused across different contexts cause a significant drop in recall and performance, so dynamic per-input estimation is needed.

Limitations

• Evaluation limited to video and mixed video-text inputs; image-only and audio settings are only analyzed qualitatively.
• Offline pattern search is needed per model, and the MM-NIAH benchmark is constructed by the authors.
• Some baseline results (e.g., VisionZip on LongVILA) are OOM, and performance drops for some tasks with mixed-modality inputs under other methods.

Experimental setup

Datasets: EgoSchema (single calibration sample, up to 25K tokens)

Benchmarks: ActivityNet-QA EgoSchema NExT-QA PerceptionTest VideoDC Video-MME V-NIAH MM-NIAH

Metrics: accuracy end-to-end latency kernel latency FLOPs recall

Baselines: Full Attention (FlashAttention-2) SparseTransformer (Fixed) SparseTransformer (Strided) A-shape Tri-shape MInference VisionZip

Builds on: MInference FlashAttention-2 A-shape Tri-shape Vertical-Slash

Closest papers by meaning

• Lifting the Veil on Visual Information Flow in MLLMs: Unlocking Pathways to Faster Inference (CVPR 2025 · similarity 0.91)
• CoVFT: Context-aware Visual Fine-tuning for Multimodal Large Language Models (CVPR 2026 · similarity 0.90)
• A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMs (CVPR 2025 · similarity 0.90)
• MHA2MLA-VLM: Enabling DeepSeek’s Economical Multi-Head Latent Attention Across Vision-Language Models (AAAI 2026 · similarity 0.90)
• SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference (ICML 2025 · similarity 0.90)
• MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.90)