Research map / Architectures and efficiency / Attention and transformers
Memory-Efficient Visual Autoregressive Modeling with Scale-Aware KV Cache Compression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
placeholder
Problem
VAR models' coarse-to-fine next-scale prediction causes KV cache to grow rapidly with scales, causing large memory consumption and latency during inference.
Approach
Layers are classified per scale as drafters (dispersed attention across history, large cache) or refiners (attention concentrated on the current token map, small cache) using an Attention Selectivity Index; cache budgets are allocated accordingly, with refiner budgets decaying across scales and the saved memory reallocated to drafters.
Key results
- At 10% budget on Infinity-8B, ScaleKV reduces KV cache from 84328 MB to 8531 MB with FID 2.12 vs 4.68 for SnapKV, and PSNR 23.25.
- On Infinity-8B with 10% cache, GenEval overall is 0.790 (vs 0.792 full) and DPG overall is 86.49 (vs 86.61 full).
- Reports up to 1.25x inference speedup on a single NVIDIA H20 GPU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ScaleKV instead of PyramidKV at a 10% cache budget | Infinity-2B · MS-COCO 2017 val | FID: 4.52 -> 2.53 | Helped |
| Use ScaleKV instead of PyramidKV at a 10% cache budget | Infinity-2B · MS-COCO 2017 val | LPIPS: 0.16 -> 0.11 | Helped |
| Use ScaleKV instead of PyramidKV at a 10% cache budget | Infinity-2B · MS-COCO 2017 val | PSNR: 20.92 -> 22.64 | Helped |
| Use ScaleKV instead of SnapKV at a 10% cache budget | Infinity-8B · MS-COCO 2017 val | FID: 4.68 -> 2.12 | Helped |
| Use ScaleKV instead of PyramidKV at a 10% cache budget | Infinity-8B · MS-COCO 2017 val | LPIPS: 0.14 -> 0.09 | Helped |
| Use ScaleKV instead of PyramidKV at a 10% cache budget | Infinity-8B · MS-COCO 2017 val | PSNR: 20.79 -> 23.25 | Helped |
| Compress the KV cache to 10% with ScaleKV | Infinity-2B · GenEval | Overall: 0.725 -> 0.730 | Helped |
| Compress the KV cache to 10% with ScaleKV | Infinity-2B · DPG | Overall: 83.06 -> 83.01 | Hurt |
| Compress the KV cache to 10% with ScaleKV | Infinity-8B · GenEval | Overall: 0.792 -> 0.790 | Hurt |
| Compress the KV cache to 10% with ScaleKV | Infinity-8B · DPG | Overall: 86.61 -> 86.49 | Hurt |
| Use ASI instead of only the Top-K ratio to identify drafters (ablation) | ScaleKV on Infinity-2B · 10% cache budget | FID: 4.60 -> 2.53 | Helped |
| Add refiner budget decay, increasing the decay rate from 0 to 70 (ablation) | ScaleKV on Infinity-2B · 10% budget, initial refiner budget 600 tokens | FID: 3.49 to 2.53 | Helped |
Surprising findings
- PyramidKV's fixed pyramid allocation sometimes performs worse than SnapKV (e.g., 4% budget on Infinity-2B FID 5.51 vs 5.10).
Limitations
- Evaluated only on Infinity-2B and Infinity-8B, the largest available VAR model, so scalability to larger models is untested.
- Post-training method that mirrors the original model's outputs, so it cannot improve a weak base model.
- Drafter/refiner identification relies on a calibration set of prompts generated by GPT-4o.
Experimental setup
Datasets: MS-COCO 2017 validation LAION-Art prompts
Benchmarks: MS-COCO 2017 GenEval DPG
Metrics: FID LPIPS PSNR GenEval overall DPG overall KV cache memory latency
Baselines: Sliding Window Attention StreamingLLM SnapKV PyramidKV
Builds on: SnapKV PyramidKV Visual Autoregressive modeling
Closest papers by meaning
- Head-Aware KV Cache Compression for Efficient Visual Autoregressive Modeling (AAAI 2026 · similarity 0.94)
- HeatKV: Head-tuned KV-cache Compression for Visual Autoregressive Modeling (NeurIPS 2026 · similarity 0.92)
- Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction (NeurIPS 2024 · similarity 0.90)
- VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Acceleration (ICLR 2025 · similarity 0.88)
- Collaborative Decoding Makes Visual Auto-Regressive Modeling Efficient (CVPR 2025 · similarity 0.88)
- HMAR: Efficient Hierarchical Masked Auto-Regressive Image Generation (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.