Research map / Architectures and efficiency / Attention and transformers
LookHere: Vision Transformers with Directed Attention Generalize and Extrapolate
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LookHere, a drop-in position encoding for plain ViTs that restricts attention heads to directional fields of view via 2D attention masks plus distance penalties, improving extrapolation to higher resolutions, and releases the ImageNet-HR test set.
Problem
Plain vision transformers extrapolate poorly to more patches at test time (higher resolutions than training) because current patch position encodings cause a distribution shift in attention heads.
Approach
Assign each attention head a direction and a fixed field of view via 2D attention masks, and penalize attention scores by Euclidean patch distance within that field, subtracted from learned attention logits before softmax; this hard-codes spatial algorithms and limits distribution shift when extrapolating.
Key results
- LH-180 outperforms 2D-RoPE by 0.93% / 1.36% on Val / v2 on best runs and by 1.64% / 1.96% on averaged runs at 224px.
- When trained at 224px and tested at 1024px on ImageNet Val, LH-45 reaches 73.15% top-1 vs 51.41% for 2D-RoPE (about 21.7 points gain as claimed in the abstract).
- LH-45 outperforms 2D-ALiBi by 9.5% on Val at 1024px.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace standard position encoding with LookHere directed attention masks and distance penalties | ViT-B/16 · ImageNet, tested at training resolution | top-1 accuracy: +1.6% avg. | Helped |
| Use LookHere instead of 2D-RoPE | ViT-B/16 · ImageNet, trained at 224² and tested at 1024² | top-1 accuracy: 21.7% | Helped |
| Add LookHere directed attention masks (ablation) | LookHere · ImageNet extrapolation | top-1 accuracy: +9.5% on Val at 1024² px | Helped |
| Remove all distance penalties (ablation) | LookHere · ImageNet extrapolation | top-1 accuracy | Hurt |
| Increase distance penalties with squared distance (ablation) | LookHere · ImageNet extrapolation | top-1 accuracy | Hurt |
| Increase distance penalties with square-root distance (ablation) | LookHere · ImageNet extrapolation | top-1 accuracy | Hurt |
| Vary the slope function (ablation) | LookHere · ImageNet | top-1 accuracy | No clear effect |
| Remove distance penalties on undirected heads (ablation) | LookHere · ImageNet | top-1 accuracy | No clear effect |
| Vary the global slope (ablation) | LookHere · ImageNet extrapolation | top-1 accuracy: 0.15% | No clear effect |
| Use different directional configurations (ablation) | LookHere · ImageNet | top-1 accuracy | No clear effect |
| Use narrower fields of view (ablation) | LookHere · ImageNet extrapolation at 1024² | top-1 accuracy | Helped |
| Tune the global slope at test resolution (ablation) | LookHere · ImageNet extrapolation at 512² | top-1 accuracy: 0.15% | No clear effect |
Surprising findings
- LH-180 averaged over hyperparameter configurations (80.01% Val) matches the best run of Steiner et al. trained for twice as long.
- Smaller objects benefit most from extrapolation, and LookHere's gains on ImageNet-HR do not come from interpolated pixels.
Limitations
- Requires hand-designed directional masks and distance penalties.
- Only ViT-B/16 evaluated; giant-scale ViTs not studied.
- Sparse-attention speedups require custom kernels not yet developed.
Experimental setup
Datasets: ImageNet-1k ADE20k Cityscapes ImageNet-S ImageNet-A ImageNet-R ImageNet v2 ImageNet ReaL
Benchmarks: ImageNet Val ImageNet ReaL ImageNet v2 ImageNet-A ImageNet-R ImageNet-HR ADE20k Cityscapes ImageNet-S
Metrics: top-1 accuracy top-5 accuracy mIoU expected calibration error adversarial accuracy under FGSM generalized Jensen-Shannon divergence
Baselines: 1D-learn 2D-sincos Factorized Fourier RPE-learn 2D-ALiBi 2D-RoPE iRPE
Builds on: 2D-RoPE ALiBi FlexiViT RPE 3-Augment
Closest papers by meaning
- You Only Need Less Attention at Each Stage in Vision Transformers (CVPR 2024 · similarity 0.88)
- Skip-Attention: Improving Vision Transformers by Paying Less Attention (ICLR 2024 · similarity 0.88)
- On the Surprising Effectiveness of Attention Transfer for Vision Transformers (NeurIPS 2024 · similarity 0.88)
- LookWhere? Efficient Visual Recognition by Learning Where to Look and What to See from Self-Supervision (NeurIPS 2025 · similarity 0.87)
- Linear Differential Vision Transformer: Learning Visual Contrasts via Pairwise Differentials (NeurIPS 2025 · similarity 0.87)
- Bridging the Divide: Reconsidering Softmax and Linear Attention (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.