Research map / Vision-language and multimodal / Multimodal LLMs

Mixing Importance with Diversity: Joint Optimization for KV Cache Compression in Large Vision-Language Models

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Attention-based visual token pruning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MixKV, a plug-and-play KV cache compression method for LVLMs that adaptively mixes per-head importance scores with key-diversity scores, weighted by each head's measured key redundancy.

Problem

KV cache growth from long multimodal sequences in LVLMs creates a memory bottleneck; existing importance-based eviction methods retain semantically redundant KV pairs and lose coverage of the full KV distribution.

Approach

Measure per-head key redundancy in closed form (average off-diagonal cosine similarity via the global mean key) and use it as an adaptive weight that interpolates between importance scores (VNorm plus attention) and a key-diversity score, so redundant heads favor diversity and less redundant heads favor importance.

Key results

• At budget 64, MixKV improves SnapKV and AdaKV by 8.0% and 9.0% average on ScreenSpot-v2 GUI grounding with Qwen2.5-VL-7B.
• MixKV improves baseline methods by an average of 5.1% across five multi-modal understanding benchmarks at extreme compression (budget 64), per the abstract.
• On Qwen3-VL-30B-A3B, MixKV boosts SnapKV on OCRBench by 13.6% at budget 64.

Surprising findings

• Adding diversity alone (s_div) substantially degrades performance relative to importance-based baselines, showing diversity cannot replace importance.
• MixKV gives slight declines on some LongBench information-localization tasks for LLMs, where head-wise redundancy is lower.

Limitations

• Evaluated only on models up to 8B in the main analysis (plus 30B MoE and 38B in additional tables); generalization to 70B+ untested.
• Head-wise redundancy is measured per sample on-the-fly, with diversity computed from key vectors only.

Experimental setup

Datasets: OCRBench

Benchmarks: DocVQA OCRBench TextVQA ChartQA TextCaps ScreenSpot-v2 LongBench

Metrics: accuracy (%) CIDEr-style captioning score average precision on GUI grounding

Baselines: SnapKV PyramidKV AdaKV SparseMM KNorm VNorm HeadKV

Builds on: SnapKV PyramidKV AdaKV SparseMM VNorm

Closest papers by meaning

• VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Acceleration (ICLR 2025 · similarity 0.93)
• EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction (NeurIPS 2026 · similarity 0.91)
• Q Cache: Visual Attention Is Valuable in Less than Half of Decode Layers for Multimodal Large Language Model (AAAI 2026 · similarity 0.90)
• PrefixKV: Adaptive Prefix KV Cache is What Vision Instruction-Following Models Need for Efficient Generation (NeurIPS 2025 · similarity 0.90)
• Prune Redundancy, Preserve Essence: Vision Token Compression in VLMs via Synergistic Importance-Diversity (ICLR 2026 · similarity 0.89)
• SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget (ICLR 2025 · similarity 0.89)