Research map / Architectures and efficiency / Attention and transformers
QT-ViT: Improving Linear Attention in ViT with Quadratic Taylor Expansion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes QT-ViT, a linear attention for ViTs that replaces softmax with a second-order Taylor expansion decomposed via Kronecker products and reduced to O(Nd^2) with a compact self-multiplication approximation, trained from scratch on ImageNet-1k without distillation.
Problem
Softmax self-attention in ViTs has quadratic time and memory cost in the number of patches, while existing linear attention variants lose accuracy, and the prior fixes (knowledge distillation, high-order residuals) raise training memory.
Approach
Approximate exp(qk/sqrt(d)) with a second-order Taylor expansion, decompose the quadratic term into a kernel dot product via Kronecker products, and keep only the self-multiplication terms x_i^2 (plus constant) so the feature map has length 2d+1, giving linear attention without distillation or masked softmax residuals.
Key results
- QT-ViT-1 reaches 79.6% top-1 at 0.52G FLOPs, versus 78.5% for first-order Taylor (ViTALiTy) and 79.4% for EfficientViT-B1 in the same kernel-ablation setup.
- QT-ViT-4 achieves 84.7% top-1 with 5.26G FLOPs, versus CSWin-B at 84.2% with 15.00G FLOPs.
- QT-ViT-6 reaches 86.0% top-1 at 27.6G FLOPs, outperforming EfficientViT-L3 (85.8%, 28.0G).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the EfficientViT attention kernel with the quadratic Taylor expansion kernel | QT-ViT · ImageNet-1k | Top-1 accuracy (%): 79.4 -> 79.6 | Helped |
| Replace the EfficientViT attention kernel with the quadratic Taylor expansion kernel | QT-ViT · COCO 2017 | AP: 39.1 -> 39.3 | Helped |
| Replace the EfficientViT attention kernel with the quadratic Taylor expansion kernel | QT-ViT · ADE20K | mIoU: 32.8 -> 33.2 | Helped |
| Use the quadratic Taylor kernel instead of the first-order Taylor kernel (ablation) | QT-ViT · ImageNet-1k | Top-1 accuracy (%): 78.5 -> 79.6 | Helped |
| Use the compact Kronecker product with self-multiplication terms instead of randomly selected quadratic terms (ablation) | QT-ViT-1 · ImageNet-1k | Top-1 accuracy (%): 61.8 -> 79.6 | Helped |
| Reduce the input dimension with pooling, p=2 (ablation) | QT-ViT-1 · ImageNet-1k | Top-1 accuracy (%): 79.7 -> 79.3 | Hurt |
| Split the input into c=2 chunks (ablation) | QT-ViT-1 · ImageNet-1k | Top-1 accuracy (%): 79.7 -> 79.4 | Hurt |
| Add absolute positional embedding (ablation) | QT-ViT · COCO 2017 | AP: 39.2 -> 39.3; 41.0 -> 41.1; 42.5 -> 42.6 | Helped |
| Use original softmax attention output during training (ablation) | QT-ViT-1 · ImageNet-1k | Top-1 accuracy (%): 79.6 -> 79.8 | Helped |
| Use original softmax attention output during training (ablation) | QT-ViT-1 · ImageNet-1k | GPU memory required per GPU during training (GB): 13.9 -> 15.8 (+13.7%) | Hurt |
Surprising findings
- Discarding the linear term of the compact Kronecker expansion does not hurt classification accuracy.
- Randomly selecting d of the d^2 quadratic terms drops QT-ViT-1 top-1 accuracy to 61.8%, far below the self-multiplication approximation (79.6%).
Limitations
- Training large models from scratch requires substantial computational resources.
- No error bars or repeated runs due to cost.
- Using the original softmax attention during training improves accuracy slightly (79.8 vs 79.6) but increases memory, so the method omits it.
Experimental setup
Datasets: ImageNet-1k COCO 2017 ADE20K
Benchmarks: ImageNet-1k COCO 2017 ADE20K
Metrics: top-1 accuracy top-5 accuracy FLOPs parameters latency AP mIoU
Baselines: EfficientViT ViTALiTy Castling-ViT FLatten Transformer Hydra Attention Swin SwinV2 FasterViT ConvNeXt CSWin EfficientNet CMT
Builds on: EfficientViT ViTALiTy Linear Transformer Castling-ViT Flatten Transformer
Closest papers by meaning
- You Only Need Less Attention at Each Stage in Vision Transformers (CVPR 2024 · similarity 0.89)
- FasterViT: Fast Vision Transformers with Hierarchical Attention (ICLR 2024 · similarity 0.88)
- Bi-ViT: Pushing the Limit of Vision Transformer Quantization (AAAI 2024 · similarity 0.88)
- GradQ-ViT: Robust and Efficient Gradient Quantization for Vision Transformers (AAAI 2025 · similarity 0.88)
- GPLQ: A General, Practical, and Lightning QAT Method for Vision Transformers (NeurIPS 2025 · similarity 0.87)
- LF-ViT: Reducing Spatial Redundancy in Vision Transformer for Efficient Image Recognition (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.