Research map / Architectures and efficiency / Pruning and efficient tuning
On the Surprising Effectiveness of Attention Transfer for Vision Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that transferring only the inter-token attention patterns from a pre-trained ViT (by copying or distilling attention maps) lets a randomly initialized student match fine-tuning accuracy on ImageNet-1K, while weaker under distribution shift.
Problem
Whether the benefit of pre-training ViTs comes from learned intra-token features or from the inter-token attention patterns that route information between tokens.
Approach
Decouple the two by transferring only attention maps from a pre-trained teacher ViT to a student that learns all other weights from scratch, either by directly copying the maps (Attention Copy) or by adding a cross-entropy distillation loss on attention maps (Attention Distillation).
Key results
- Attention Distillation with a MAE ViT-L teacher reaches 85.7 top-1 on ImageNet-1K, matching full fine-tuning (85.7) and above scratch (83.0).
- Attention Copy reaches 85.1, recovering 77.8% of the gap between scratch and fine-tuning.
- Ensembling Attention Distillation with the fine-tuned MAE reaches 86.3 (+0.6 over fine-tuned); Attention Copy with a FLIP teacher reaches 86.6.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| transfer MAE attention maps instead of training from scratch | ViT-L on ImageNet-1K · ImageNet-1K | accuracy: 83.0 -> 85.1 | Helped |
| distill MAE attention maps rather than copy them | ViT-L on ImageNet-1K · ImageNet-1K | accuracy: 85.1 -> 85.7 | Helped |
| ensemble Attention Distillation with fine-tuned MAE | ViT-L on ImageNet-1K · ImageNet-1K | accuracy: +0.6 | Helped |
| transfer queries Q rather than attention maps (ablation) | Attention Copy, ViT-L · ImageNet-1K | accuracy: 85.1 -> 85.6 | Helped |
| transfer keys K or values V rather than attention maps (ablation) | Attention Copy, ViT-L · ImageNet-1K | accuracy: 85.1 -> 84.4 | Hurt |
| distill queries Q instead of attention maps (ablation) | Attention Distillation, ViT-L · ImageNet-1K | accuracy: 85.7 -> 81.3 | Hurt |
| transfer more attention layers, particularly later layers (ablation) | Attention Copy, ViT-L · ImageNet-1K | accuracy | Helped |
| transfer more attention heads (ablation) | Attention Copy, ViT-L · ImageNet-1K | accuracy | Helped |
| average attention maps across examples before transfer (ablation) | Attention Copy, ViT-L · ImageNet-1K | accuracy: 85.1 -> 79.7 | Hurt |
| average attention maps across query tokens before transfer (ablation) | Attention Copy, ViT-L · ImageNet-1K | accuracy: 85.1 -> 79.9 | Hurt |
| distill residual-stream features instead of attention maps (ablation) | ViT-L on ImageNet-1K · ImageNet-1K | accuracy: 85.7 -> 81.3 | Hurt |
| use a randomly initialized teacher for attention transfer (ablation) | ViT-L attention transfer · ImageNet-1K | accuracy: 85.1 -> 72.7 | Hurt |
Surprising findings
- Transferring only query matrices Q (85.6) outperforms transferring the full attention map (85.1) under Attention Copy.
- Attention transfer from a randomly initialized teacher drops to 72.7, but pre-trained attention from a FLIP teacher gives 86.6, exceeding the MAE teacher.
Limitations
- Attention transfer underperforms fine-tuning when pre-training and downstream datasets differ (e.g., copy drops 2.1 with ImageNet-22K/COCO pre-training and on iNaturalist).
- Attention Distillation loses the out-of-distribution effective robustness of fine-tuned MAE (e.g., ImageNet-A 54.3 vs 56.5).
- Attention Distillation requires the architecture to compute attention maps, so it does not directly apply to ConvNets or other architectures.
Experimental setup
Datasets: ImageNet-1K ImageNet-22K COCO iNaturalist 2017 iNaturalist 2018 iNaturalist 2019 LAION-2B
Benchmarks: ImageNet-1K ImageNet-A ImageNet-R ImageNet-S ImageNet-V2 iNaturalist 2017 iNaturalist 2018 iNaturalist 2019 COCO object detection
Metrics: top-1 accuracy AP box AP mask CKA ensemble accuracy
Baselines: scratch training full weight fine-tuning feature distillation Q distillation MoCo-v3 DINO
Builds on: Vision Transformer Masked Autoencoders knowledge distillation layerwise learning rate decay
Closest papers by meaning
- Attention Transfer Is Not Universally Effective for Vision Transformers (NeurIPS 2026 · similarity 0.94)
- Improving Adversarial Transferability on Vision Transformers via Forward Propagation Refinement (CVPR 2025 · similarity 0.89)
- ScaleKD: Strong Vision Transformers Could Be Excellent Teachers (NeurIPS 2024 · similarity 0.88)
- Vision Transformers Need Registers (ICLR 2024 · similarity 0.88)
- You Only Need Less Attention at Each Stage in Vision Transformers (CVPR 2024 · similarity 0.88)
- ExPLoRA: Parameter-Efficient Extended Pre-Training to Adapt Vision Transformers under Domain Shifts (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.