Research map / Architectures and efficiency / Attention and transformers
PADRe: A Unifying Polynomial Attention Drop-in Replacement for Efficient Vision Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PADRe, a unifying polynomial attention drop-in replacement built from Hadamard-product polynomial terms, showing Hyena, Mamba, SimA, Conv2Former and Castling-ViT are special cases while matching self-attention accuracy at linear cost on vision tasks.
Problem
Self-attention in vision transformers has quadratic cost in token count, which is prohibitive for high-resolution images, 3D point clouds, and resource-constrained mobile hardware; many alternative attentions lack a unifying view and often degrade accuracy as drop-in replacements.
Approach
Express attention replacement as a multivariate polynomial of the input built recursively from linear token/channel mixings combined with Hadamard products, giving linear time and memory in tokens for fixed degree, with single-head operation; many existing attention variants are shown to be special cases.
Key results
- PADRe-3 on DeiT-Tiny reaches 76.5% top-1 vs 72.2% for standard self-attention on ImageNet-1K.
- On nuScenes, PADRe-2 in DSVT matches NDS (71.1) with 28% fewer backbone FLOPs (10.7G vs 14.9G).
- At 4096 tokens, PADRe-2 attention latency is 0.01 ms on GPU vs 0.43 ms for self-attention; on NPU 1.62 ms vs 68.60 ms (reported speedups of 11x-43x).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace self-attention with degree-3 PADRe | DeiT-Tiny · ImageNet-1K | Top-1 accuracy: 72.2 -> 76.5 | Helped |
| Replace self-attention with degree-2 PADRe | DeiT-Small · ImageNet-1K | Top-1 accuracy: 79.8 -> 80.4 | Helped |
| Replace self-attention with degree-2 PADRe | DeiT-Base · ImageNet-1K | Top-1 accuracy: 81.8 -> 81.4 | Hurt |
| Replace DETR encoder self-attention with degree-2 PADRe | DETR · COCO | AP: 40.6 -> 40.2 | No clear effect |
| Replace self-attention with degree-2 PADRe | DSVT · nuScenes validation | NDS: 71.1 -> 71.1 | No clear effect |
| Replace self-attention with degree-2 PADRe | DSVT · nuScenes validation | mAP: 66.4 -> 66.0 | Hurt |
| Increase polynomial degree from 2 to 3 (ablation) | PADRe in DeiT-Tiny · ImageNet-1K | Top-1 accuracy: 74.5 -> 76.5 | Helped |
| Increase polynomial degree from 3 to 4 (ablation) | PADRe in DeiT-Tiny · ImageNet-1K | Top-1 accuracy: 76.5 -> 76.9 | Helped |
| Increase polynomial degree from 2 to 3 (ablation) | PADRe attention · 4096 tokens | Latency on GPU: 0.01 -> 0.02 | Hurt |
| Increase polynomial degree from 2 to 3 (ablation) | PADRe attention · 4096 tokens | Latency on NPU: 1.62 -> 3.17 | Hurt |
| Increase polynomial degree from 2 to 4 (ablation) | PADRe attention · 4096 tokens | Latency on GPU: 0.01 -> 0.04 | Hurt |
| Increase polynomial degree from 2 to 4 (ablation) | PADRe attention · 4096 tokens | Latency on NPU: 1.62 -> 4.22 | Hurt |
Surprising findings
- Hydra attention and linear-angular attention lose accuracy when used as drop-in replacements.
- Performance saturates beyond degree 3 (PADRe-4 only 76.9% vs 76.5%), with the authors hypothesizing numerical stability of monomials as a cause.
Limitations
- Accuracy saturates beyond degree 3, with unresolved cause; monomial basis may be numerically unstable.
- Only self-attention replacement evaluated; cross-attention and multi-modal settings not empirically tested.
- Some baseline comparisons rely on numbers reported in other papers and missing entries (e.g., NPU latency for alternatives).
Experimental setup
Datasets: ImageNet-1K COCO nuScenes
Benchmarks: ImageNet-1K COCO nuScenes
Metrics: top-1 accuracy AP NDS mAP FLOPs latency
Baselines: DeiT Hydra attention Efficient attention Linear-angular attention Enhanced linear attention k-NN attention Focused linear attention Vision Mamba SimA Hyena DETR DSVT
Builds on: DeiT DETR DSVT Hyena Mamba
Closest papers by meaning
- Orthogonal Transformer: An Efficient Vision Transformer Backbone with Token Orthogonalization (NeurIPS 2022 · similarity 0.88)
- Skip-Attention: Improving Vision Transformers by Paying Less Attention (ICLR 2024 · similarity 0.88)
- Breaking the Low-Rank Dilemma of Linear Attention (CVPR 2025 · similarity 0.87)
- You Only Need Less Attention at Each Stage in Vision Transformers (CVPR 2024 · similarity 0.87)
- Bridging the Divide: Reconsidering Softmax and Linear Attention (NeurIPS 2024 · similarity 0.87)
- MonarchAttention: Zero-Shot Conversion to Fast, Hardware-Aware Structured Attention (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.