Research map / Architectures and efficiency / Attention and transformers
AVQ-Attention: Adaptive Vector-Quantized Attention
Winfried van den Dool, Patrick Forré, Amirhossein Habibian, Yuki Asano, Max Welling
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace flat VQ codebook with adaptive parent-child refinement | VQ-attention · ADE20K (DPT-L, 30-epoch comparison) | mIoU: 42.70 -> 43.33 (vs flat M=128) | Improved |
| fuse VQ precompute into a single kernel (ablation) | VQ-attention · Synthetic N=65k kernel benchmark | kernel time (ms): 20.7 -> 10.6 | Improved |
| replace Flash Attention with fused AVQ-attention | attention kernel · Synthetic N=65k kernel benchmark | kernel time (ms): 847 -> 6.67 | Improved |
| replace Flash Attention with AVQ-attention at short sequence | attention kernel · Synthetic N=1k kernel benchmark | kernel time (ms): 0.21 -> 0.13 (AVQ 64/8/8); 0.21 -> 0.21 (128/16/8) | Mixed |
| replace attention with AVQ in SD1.5 UNet (level-0 blocks) | SD1.5 · SD1.5 COCO 1024^2 (ScaleCrafter) | FID: 41.03 -> 35.39 | Improved |
| replace attention with AVQ in SD1.5 UNet (level-0 blocks) | SD1.5 · SD1.5 COCO 512^2 | FID: 12.75 -> 12.50 | No clear effect |
| also replace level-1 (N=1k) attention blocks with AVQ (ablation) | AVQ M64P16C8 · SD1.5 COCO 1024^2 (ScaleCrafter) | UNet time per step (ms): 135.06 -> 128.59 ms, FID 35.65 -> 36.29 | Mixed |
| reduce refined parents P at inference from 32 to 16 (ablation) | AVQ M0=64 C=8 · ImageNet-1k (ViT-B elastic inference, M0=64 C=8) | top-1 accuracy: 82.29 -> 82.04 | No clear effect |
| reduce refined parents P at inference from 8 to 4 (ablation) | AVQ M0=64 C=8 · ImageNet-1k (ViT-B elastic inference, M0=64 C=8) | top-1 accuracy: 81.16 -> 79.23 | Hurt |
| use tiled approximate denominator for importance instead of exact (ablation) | AVQ importance scores · ImageNet (ViT-Base), P=32 | attention mass captured: 82.0% -> 77.7% | Hurt |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 0 of 1 on most shared results.
- Flash Attention-v2 — FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (NeurIPS 2022) · better on 1 of 2 shared results
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration (ICLR 2025 · similarity 0.89)
- Token Statistics Transformer: Linear-Time Attention via Variational Rate Reduction (ICLR 2025 · similarity 0.88)
- A-VL: Adaptive Attention for Large Vision-Language Models (AAAI 2025 · similarity 0.88)
- Transformer-VQ: Linear-Time Transformers via Vector Quantization (ICLR 2024 · similarity 0.88)
- Delta Attention: Fast and Accurate Sparse Attention Inference by Delta Correction (NeurIPS 2025 · similarity 0.87)
- MVA: Linear Attention with High-order Query-Keys Integration and Multi-level Vocabulary Decomposition (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
