Research map / Architectures and efficiency / Attention and transformers
Transformer-VQ: Linear-Time Transformers via Vector Quantization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Transformer-VQ is a decoder-only transformer that computes softmax dense self-attention in linear time by vector-quantizing keys and using a compressive block cache, reaching competitive results on Enwik8, PG-19, and ImageNet64.
Problem
Standard transformer self-attention is quadratic in sequence length, limiting scaling to very long sequences in autoregressive modeling.
Approach
Vector-quantize the keys with a straight-through estimator so that softmax attention over quantized keys can be factored through a shortcode-indexed compressive cache, giving exact dense attention over quantized keys computed in linear time with block-level recurrences.
Key results
- Transformer-VQ obtains 0.99 bpb on Enwik8, matching Transformer-XL while using 33% fewer parameters.
- Transformer-VQ reaches 26.6 word-level perplexity on PG-19, close to Block-Recurrent Transformer's 26.5.
- The 1.2B Transformer-VQ achieves 3.16 bpb on ImageNet64, a new state of the art; the optimized implementation is over 12x faster than full attention at sequence length 32k.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the codebook size from S = 256 to S = 1024 (ablation) | Transformer-VQ · Enwik8 validation | BPB: 1.010 -> 1.000 | Helped |
| Increase the codebook size from S = 256 to S = 1024 (ablation) | Transformer-VQ · Enwik8 validation | Latency (Rel.): 0.927 -> 1.109 | Hurt |
| Add the compressive cache (ablation) | Transformer-VQ · Enwik8 validation | BPB: 1.026 -> 1.010 | Helped |
| Add the compressive cache (ablation) | Transformer-VQ · Enwik8 validation | Latency (Rel.): 0.836 -> 0.927 | Hurt |
| Use Transformer-VQ instead of Full Attention | SHGA · Sequence length 8192 | tokens/sec: 23.9k -> 77.2k | Helped |
| Use Transformer-VQ instead of Full Attention | SHGA · Sequence length 32768 | tokens/sec: 6.5k -> 82.1k | Helped |
Surprising findings
- Removing the compressive cache speeds training only modestly (about 1.1x) but significantly degrades Enwik8 BPB, and Transformer-VQ falls short of state-of-the-art on Enwik8 despite strong results elsewhere.
Limitations
- Falls short of state-of-the-art on Enwik8 (best reported models reach 0.94 BPB).
- Overfitting was significant on Enwik8, and i.i.d. attention dropout could not be used due to the compressive cache.
- Largest models evaluated only up to 1.3B parameters; formal scaling laws left to future work.
Experimental setup
Datasets: Enwik8 PG-19 ImageNet64
Benchmarks: Enwik8 PG-19 ImageNet64
Metrics: bits-per-byte word-level perplexity training throughput (tokens/sec) latency
Baselines: Transformer-XL Sparse Transformer Compressive Transformer Block-Recurrent Transformer Perceiver AR MegaByte Efficient VDVAE Full attention (quadratic)
Builds on: Transformer-XL FLASH Vector-quantized representation learning (VQ-VAE) Gated attention unit
Closest papers by meaning
- MimiQ: Low-Bit Data-Free Quantization of Vision Transformers with Encouraging Inter-Head Attention Similarity (AAAI 2025 · similarity 0.88)
- MonarchAttention: Zero-Shot Conversion to Fast, Hardware-Aware Structured Attention (NeurIPS 2025 · similarity 0.87)
- Token Statistics Transformer: Linear-Time Attention via Variational Rate Reduction (ICLR 2025 · similarity 0.87)
- Finite Scalar Quantization: VQ-VAE Made Simple (ICLR 2024 · similarity 0.86)
- Bridging the Divide: Reconsidering Softmax and Linear Attention (NeurIPS 2024 · similarity 0.86)
- Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators? (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.