Research map / Vision-language and multimodal / Multimodal LLMs

TGV-KV: Text-Grounded KV Eviction for Vision-Language Models

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Attention-based visual token pruning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TGV-KV, a text-grounded KV cache eviction method for VLMs that allocates layer budgets via text-vision attention, ranks vision KV by text-weighted attention, and prioritises retaining text KV.

Problem

KV cache eviction methods designed for LLMs degrade badly on VLMs because of the modality gap between vision and text tokens (vision tokens are redundant, text tokens are sensitive to eviction, and cross-modal attention is weak), causing large accuracy loss at low KV retention budgets.

Approach

Ground KV importance in textual guidance: layer budgets follow normalized text-vision attention mass (TVB), vision token scores are a text-weighted sum of text-vision attention with weights from text-text attention (TWR), and eviction only removes text KV once the budget cannot hold all text (TPR).

Key results

• On LLaVA-NeXT at 5% retention, TGV-KV keeps 99.2% of full-KV accuracy on VizWiz and 97.4% on TextVQA.
• On Qwen3-VL-4B at 5% retention, TGV-KV preserves 93.3% of DocVQA and 92.1% of VizWiz performance.
• At 16k context with 5% KV, throughput rises 52.6% over vanilla on LLaVA-1.5-7B (A800), with KV memory reduced by about 95%.

Surprising findings

• Evicting text tokens first collapses performance (e.g., ChartQA near 0.2 at 50% retention), far worse than evicting vision tokens.
• Text-only eviction-based performance on some text-dominant tasks slightly exceeds the vanilla model, which the authors attribute to filtering irrelevant information.

Limitations

• Evaluated only in a training-free setting with a limited set of VLMs and eviction baselines.
• Head-wise budget allocation is not considered because it would break attention parallelism.
• When text is long, all vision KV may be evicted by the TPR policy, which the authors note as a regime needing scrutiny.

Experimental setup

Datasets: ChartQA DocVQA VizWiz TextVQA TextCaps COCO-Caption-2017 Video-TT POPE ScienceQA MME GQA

Benchmarks: ChartQA DocVQA VizWiz TextVQA TextCaps COCO-Caption-2017 Video-TT MME POPE GQA

Metrics: relaxed accuracy ANLS accuracy CIDEr ROUGE-L BLEU METEOR memory (GB) latency (ms) throughput (tokens/s)

Baselines: StreamingLLM SnapKV H2O ElasticCache PrefixKV AirCache DivPrune VisionZip VisPruner CDPruner

Builds on: SnapKV H2O PyramidKV PrefixKV StreamingLLM

Closest papers by meaning

• VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Acceleration (ICLR 2025 · similarity 0.92)
• PrefixKV: Adaptive Prefix KV Cache is What Vision Instruction-Following Models Need for Efficient Generation (NeurIPS 2025 · similarity 0.91)
• MHA2MLA-VLM: Enabling DeepSeek’s Economical Multi-Head Latent Attention Across Vision-Language Models (AAAI 2026 · similarity 0.90)
• LLMC+: Benchmarking Vision-Language Model Compression with a plug-and-play Toolkit (AAAI 2026 · similarity 0.90)
• A-VL: Adaptive Attention for Large Vision-Language Models (AAAI 2025 · similarity 0.90)
• MMTok: Multimodal Coverage Maximization for Efficient Inference of VLMs (ICLR 2026 · similarity 0.90)