Research map / Vision-language and multimodal / Multimodal LLMs
A-VL: Adaptive Attention for Large Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes A-VL, a training-free, plug-and-play adaptive attention method that manages KV cache separately for vision tokens (core/secondary/minor sets with periodic updates, prefill pruning) and text tokens (H2O-style accumulated attention eviction) in LVLMs.
Problem
LVLM inference is expensive because the KV cache grows with long visual token sequences; existing adaptive attention methods (StreamingLLM, H2O, FastV) are designed for single-modal LLMs or break when combined with KV cache.
Approach
Observes that vision and text tokens in LVLMs have different attention patterns (vision attention is sparse, drifts slowly, and stays high; text attention decays rapidly), and manages each modality's cache separately: vision tokens are stored in a secondary set but only core tokens are computed, refreshed every K steps; text tokens use H2O-style local plus heavy-hitter eviction.
Key results
- With only 50% of the KV cache, A-VL averages 58.93 vs 58.78 original on LLaVA-1.5 7B across six benchmarks, while FastV drops to 57.12 and H2O to 58.54.
- On LLaVA-1.6 7B, retaining ~45% stored and ~35% used cache keeps average near-lossless (71.28 vs 71.47 original).
- Decoder inference latency on LLaVA-1.6 7B is 50.5% of original at 50% cache compression, with 1.8x speedup from compression and 1.1x from the custom CUDA operator.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use A-VL instead of H2O | LLaVA 1.5 7B · Six-task average | average score: 58.54 -> 58.93 | Helped |
| Use A-VL instead of H2O | LLaVA 1.5 13B · Six-task average | average score: 62.18 -> 62.65 | Helped |
| Use A-VL instead of H2O | LLaVA 1.6 7B · Six-task average | average score: 70.75 -> 71.28 | Helped |
| Use A-VL instead of H2O | LLaVA 1.6 13B · Six-task average | average score: 71.90 -> 72.22 | Helped |
| Use A-VL instead of H2O | Qwen-VL · Six-task average | average score: 61.77 -> 62.39 | Helped |
| Add core-cache selection and update the core set every K=3 steps (ablation) | A-VL on LLaVA 1.5 7B · DocVQA | score: 31.26 -> 31.02 | Hurt |
| Add prefill image-token retention P=90 (ablation) | A-VL on LLaVA 1.5 7B · DocVQA | score: 31.02 -> 31.02 | No clear effect |
| Add text-cache retention T=50 (ablation) | A-VL on LLaVA 1.5 7B · DocVQA | score: 31.02 -> 30.92 | Hurt |
| Retain 40% of image cache as secondary cache (ablation) | A-VL on LLaVA 1.5 7B · DocVQA | score | No clear effect |
| Retain less than 80% of image tokens in prefill (ablation) | A-VL on LLaVA 1.5 7B · DocVQA | score | Hurt |
| Increase text-cache retention above 60% (ablation) | A-VL · DocVQA | score | Helped |
| Compress KV cache to 50% | A-VL on LLaVA 1.6 7B · Decoder inference | speedup: 1.8x | Helped |
Surprising findings
- FastV, when its pruned tokens are reused via KV cache, degrades noticeably (e.g., Qwen-VL average falls from 62.66 to 55.92), since evicted tokens cause cache misses.
- Image caption tasks tolerate reducing image tokens with little loss, whereas OCR and VQA require finer detail and degrade more under pruning.
Limitations
- Gains over H2O are modest on several benchmarks; memory savings are reported at batch size one.
- Custom CUDA operator speedup is smaller in end-to-end PyTorch than in isolated tests.
- Evaluated only on LLaVA and Qwen-VL families and three task categories; excludes multiple-choice benchmarks like SEED-Bench by design.
Experimental setup
Datasets: Flickr30k OCRVQA DocVQA TextVQA VQAv2 Nocaps OCRBench
Benchmarks: DocVQA TextVQA OCRBench Nocaps Flickr30k VQAv2
Metrics: ANLS accuracy CIDEr OCRBench score KV cache memory decoder latency
Baselines: FastV H2O Original (full KV cache)
Builds on: H2O FastV StreamingLLM
Closest papers by meaning
- Attention-aware Inference Optimizations for Large Vision-Language Models with Memory-efficient Decoding (CVPR 2026 · similarity 0.92)
- A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMs (CVPR 2025 · similarity 0.92)
- Large Vision-Language Models Get Lost in Attention (ICML 2026 · similarity 0.91)
- VISion On Request: Enhanced VLLM efficiency with sparse, dynamically selected, vision-language interactions (CVPR 2026 · similarity 0.91)
- MHA2MLA-VLM: Enabling DeepSeek’s Economical Multi-Head Latent Attention Across Vision-Language Models (AAAI 2026 · similarity 0.91)
- Conical Visual Concentration for Efficient Large Vision-Language Models (CVPR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.