Research map / Vision-language and multimodal / Multimodal LLMs

Attention-aware Inference Optimizations for Large Vision-Language Models with Memory-efficient Decoding

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Attention-based visual token pruning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AttentionPack, which compresses VLM KV caches via multi-head SVD over visual tokens and attention-aware partial decompression, reducing cache size up to ~8x with minimal quality loss and faster batch decoding.

Problem

KV cache memory overhead during decoding limits batch size and context length for large vision-language models with many visual tokens, causing high inference latency.

Approach

Exploits the low-rank structure of key/value caches for visual tokens by jointly compressing across heads via SVD along the hidden dimension (no token eviction), and decompresses only high-importance tokens at full rank using moving-average attention statistics, with lower ranks for less-attended tokens.

Key results

• On LLaVA1.5-7B, AttentionPack (rank 32) achieves 7.24x cache reduction with +65% average throughput change, while rank 64 gives 5.09x with accuracy within ~0.3% of full KV caching on A-OKVQA and OCR-VQA.
• On VideoLLaVA-7B, AttentionPack achieves 8.11x smaller cache with +60% throughput and performance within 0.4% of FastV on MSVD-QA and MSRVTT-QA.
• Fused kernel implementation almost halves decoding latency for 32 tokens in single and batch inference compared to standard implementation on LLaVA1.5-7B.

Surprising findings

• Compressing textual and visual tokens together caused around 20% loss on A-OKVQA, so compression is applied only to visual tokens.
• Compression sometimes slightly improved performance on OCR-VQA (e.g., +1.39% at rank 64), suggesting it filters irrelevant visual information.

Limitations

• Single-instance inference latency can increase by up to 30% due to decompression overhead.
• Eviction combined with the method hurts OCR-VQA performance because needed visual tokens change at each step.
• Textual-token compression evidence is limited to a small LongBench experiment.

Experimental setup

Datasets: OCR-VQA A-OKVQA MMMU MSVD-QA MSRVTT-QA LongBench MTBench

Benchmarks: A-OKVQA OCR-VQA MMMU MSVD-QA MSRVTT-QA LongBench

Metrics: accuracy ROUGE-L F1 cache size reduction throughput change decoding latency

Baselines: Full KV caching FastV ScissorHands H2O MiniCache KVQuant

Builds on: FlashAttention H2O FastV KVQuant GQA

Closest papers by meaning

• A-VL: Adaptive Attention for Large Vision-Language Models (AAAI 2025 · similarity 0.92)
• Q Cache: Visual Attention Is Valuable in Less than Half of Decode Layers for Multimodal Large Language Model (AAAI 2026 · similarity 0.91)
• Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsification (ICLR 2025 · similarity 0.90)
• Large Vision-Language Models Get Lost in Attention (ICML 2026 · similarity 0.90)
• VISion On Request: Enhanced VLLM efficiency with sparse, dynamically selected, vision-language interactions (CVPR 2026 · similarity 0.90)
• MHA2MLA-VLM: Enabling DeepSeek’s Economical Multi-Head Latent Attention Across Vision-Language Models (AAAI 2026 · similarity 0.90)