Research map / Architectures and efficiency / Attention and transformers
Window Attention is Bugged: How not to Interpolate Position Embeddings
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Identifies a bug where interpolating absolute position embeddings combined with window attention degrades high-resolution finetuning, and proposes "absolute win" (tiled window embeddings plus global embeddings) to fix it in Hiera and ViTDet, yielding HieraDet with 61.7 COCO box mAP.
Problem
Naively bicubic-interpolating absolute position embeddings in vision transformers that also use window attention misaligns the learned embeddings with windows, hurting accuracy when finetuning at higher resolution (for Hiera and ViTDet).
Approach
Replace naive interpolation with an "absolute win" position embedding: a window embedding that is tiled to the full feature map plus a separately learned global embedding that is interpolated, so the embedding stays aligned with window attention at any image size; for ViTDet, tile the pretrained embedding and use relpos only in global layers.
Key results
- Naive absolute embeddings cause Hiera-L top-1 accuracy to drop 0.4% when finetuned from 224px to 256px on ImageNet-1k; absolute win fixes this.
- Absolute win speeds up ViTDet by up to 43% while increasing mAP, and HieraDet with absolute win reaches 61.7 box mAP on COCO with ImageNet-1k pretraining only.
- Absolute win gives up to +1.5 box AP and +1.6 mask AP over the original Hiera detection results, and Hiera-B outperforms HTC-based methods with heavier backbones.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace naive position-embedding interpolation with absolute win | Hiera-L · ImageNet-1k, finetuning from 224px to 256px | top-1 accuracy: 0.4% | Helped |
| Use absolute win instead of original absolute embeddings | Fully supervised Hiera-B · ImageNet-1k | accuracy | Helped |
| Use absolute win | Fully supervised Hiera-L · ImageNet-1k | accuracy: +3.7% | Helped |
| Reset position embeddings and train supervised for 50 epochs (ablation) | MAE-pretrained Hiera-L · ImageNet-1k | accuracy: 84.1% v.s. 85.6% | Hurt |
| Use absolute win rather than naive interpolation | Hiera · ImageNet-1k at larger finetuning resolutions | accuracy | Helped |
| Use absolute win at high resolution | Hiera-L · Kinetics-400 | top-1 accuracy: 1-2% | Helped |
| Use absolute win for high-resolution Kinetics-400 pretraining | Hiera-L and Hiera-H · AVA v2.2 | AP: +2.6 AP for L and +1.3 AP for H | Helped |
| Tile pretrained position embeddings and use absolute win without relative position embeddings | ViTDet · COCO | box mAP: +0.4 mAP box | Helped |
| Tile pretrained position embeddings and use absolute win without relative position embeddings | ViTDet · COCO | mask mAP: +0.6 mAP mask | Helped |
| Remove relative position embeddings during finetuning (ablation) | Original ViTDet · COCO | box mAP: up to 1 mAP box | Hurt |
| Remove relative position embeddings during finetuning (ablation) | Original ViTDet · COCO | mask mAP: 0.7 mAP mask | Hurt |
| Add a zero-initialized absolute position embedding at the input or qkv outputs of global-attention blocks (ablation) | ViTDet absolute win · COCO | detection performance | No clear effect |
Surprising findings
- Relative position embeddings in ViTDet partly mask the bug; removing them during finetuning drops performance, and absolute win without relpos recovers gains.
- Absolute win improves fully supervised Hiera-L on ImageNet-1k by 3.7%, suggesting supervised pure transformers learn suboptimal position embeddings.
Limitations
- Some hyperparameters (e.g., for high-resolution H models) were not tuned and may be suboptimal; H models used batch size 1 per GPU and torch SDPA which may affect accuracy.
- Only two architectures (Hiera and ViTDet) are studied in depth; effects on other models like Swin are suggested but not tested.
Experimental setup
Datasets: ImageNet-1k Kinetics-400 AVA v2.2 COCO train2017 LVIS
Benchmarks: ImageNet-1k Kinetics-400 AVA v2.2 COCO val2017 LVIS
Metrics: top-1 accuracy box mAP mask mAP AP inference ms/im
Baselines: Hiera ViTDet MViTv2 Swin VideoMAE HTC
Builds on: Hiera ViTDet Mask R-CNN MAE Detectron2
Closest papers by meaning
- LookHere: Vision Transformers with Directed Attention Generalize and Extrapolate (NeurIPS 2024 · similarity 0.85)
- Maximizing the Position Embedding for Vision Transformers with Global Average Pooling (AAAI 2025 · similarity 0.84)
- Configuring Data Augmentations to Reduce Variance Shift in Positional Embedding of Vision Transformers (AAAI 2025 · similarity 0.83)
- Win-Win: Training High-Resolution Vision Transformers from Two Windows (ICLR 2024 · similarity 0.83)
- Attention Guided CAM: Visual Explanations of Vision Transformer Guided by Self-Attention (AAAI 2024 · similarity 0.83)
- Register and [CLS] tokens induce a decoupling of local and global features in large ViTs (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.