Research map / Vision-language and multimodal / Multimodal LLMs

Q Cache: Visual Attention Is Valuable in Less than Half of Decode Layers for Multimodal Large Language Model

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Training-free token reduction for MLLMs · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Lazy Attention with a Q Cache that shares queries (and thus attention patterns) across adjacent decode layers in MLLMs, cutting KV cache by over 35% and raising throughput ~1.5x with about 1% accuracy loss.

Problem

MLLM inference is expensive because of many visual tokens, which inflate attention compute and KV cache; existing token-wise pruning methods are incompatible with Flash Attention and damage KV cache integrity, hurting long generation such as captioning.

Approach

Using JS divergence of last-row attention, adjacent decode layers with similar attention are grouped into lazy blocks; non-first layers reuse the first layer's query (Q Cache) and its keys, with GLA sharing all tokens and VLA sharing only visual tokens. This is a layer-wise, training-free-style reduction orthogonal to token pruning.

Key results

• VLA reduces KV cache by about 35% and achieves 1.5x throughput on LLaVA-NEXT-7B, with GLA reaching 1.6x throughput.
• VLA on LLaVA-v1.5-7B matches or beats token-pruning baselines (e.g., TextVQA 45.70 vs FastV 45.52, COCO 110.51 vs 110.48 for FastV) while retaining near-vanilla captioning scores.
• Combining VLA with FastV loses only about 1-2 points on most metrics (e.g., TextVQA 45.70 to 44.13).

Surprising findings

• Attention similarity between adjacent decode layers is high enough that over 50% of layers can share attention, and LLM (Vicuna) alone shows the same lazy pattern.
• Applying the method directly on Vicuna during LLaVA instruction tuning gives results comparable to post-training an already-tuned LLaVA.

Limitations

• GLA consistently underperforms VLA, especially on text-heavy tasks such as MMMU and MMBench.
• VLA leaves parameter count unchanged, so it does not reduce model parameter memory.
• Layer grouping relies on a similarity threshold and empirical analysis rather than a principled selection criterion.

Experimental setup

Datasets: LLaVA instruction tuning data

Benchmarks: GQA VQAv2 TextVQA VizWiz AI2D ScienceQA-IMG MMMU MMBench POPE COCO Caption NoCaps Flickr30K

Metrics: accuracy CIDEr FLOPs latency KV cache memory throughput

Baselines: FastV VTW HiRED PruMerge+ SparseVLM

Builds on: FastV Flash Attention Mixture of Experts

Closest papers by meaning

• Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference (AAAI 2025 · similarity 0.91)
• HAWK: Head Importance-Aware Visual Token Pruning in Multimodal Models (CVPR 2026 · similarity 0.91)
• ST3: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimming (AAAI 2025 · similarity 0.91)
• VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Acceleration (ICLR 2025 · similarity 0.91)
• Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings (NeurIPS 2025 · similarity 0.91)
• Attention-aware Inference Optimizations for Large Vision-Language Models with Memory-efficient Decoding (CVPR 2026 · similarity 0.91)