Research map / Architectures and efficiency / Attention and transformers

Attention Transfer Is Not Universally Effective for Vision Transformers

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Efficient and robust vision transformers · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Revisits Attention Transfer for ViTs across 20 teachers from 11 families and finds it fails for 4 families (DINOv2, DINOv2-wr, CLIP, BEiTv2), which a native-architecture component addition (LayerScale, PreLayerNorm, RelativePositionBias) reverses.

Problem

Prior work claims that transferring only attention patterns from a pre-trained ViT teacher to a randomly initialized standard student recovers the full benefit of the pre-trained weights; it is unclear whether this holds across diverse ViT families.

Approach

Shows Attention Transfer is family-dependent: 4 families consistently underperform from-scratch training under a standard student, and the failure is attributed to architectural mismatch (missing teacher-native components), which when added in random init flips transfer to positive gains.

Key results

• Attention Transfer fails for 4 families (DINOv2, DINOv2-wr, CLIP, BEiTv2), with the worst case DINOv2-wr-B 5.1% below the No-Transfer baseline under Attention Copy.
• Adding teacher-native components (LayerScale, PreLayerNorm, RelativePositionBias) in random init flips the failure to positive gains for all 4 failure families under both transfer methods.
• Native-architecture No-Transfer baselines are worse than standard-arch ones (e.g., DINOv2-S 60.1 vs 63.1), so the components alone do not boost from-scratch training.

Surprising findings

• Native-architecture from-scratch training is worse than standard-architecture from-scratch training for all failure families.
• Neither the choice of transfer loss (CE, MSE, JSD, L1) nor common pre-training recipe properties explain the family-level failure.

Limitations

• Scope limited to softmax-to-softmax attention transfer with matched-size teacher-student pairs and classification-centered tasks.
• Mechanistic evidence is empirical; no formal theory of why architectural mismatch matters.
• 20-epoch training used for main benchmark, relying on the claim that it reveals the boundary; DINOv2 patch size was interpolated from 14 to 16.

Experimental setup

Datasets: ImageNet-1K iNaturalist 2017 iNaturalist 2018 iNaturalist 2019 ImageNet-A ImageNet-R ImageNet-Sketch ImageNet-V2

Benchmarks: ImageNet-1K iNaturalist 2017 iNaturalist 2018 iNaturalist 2019 ImageNet-A ImageNet-R ImageNet-Sketch ImageNet-V2

Metrics: Top-1 accuracy Top-1 accuracy gap vs No-Transfer KL divergence JS divergence

Baselines: No-Transfer (from-scratch ViT) Attention Copy Attention Distillation

Builds on: Attention Transfer (prior work on attention-only ViT transfer) CaiT LayerScale DINOv2 BEiTv2

Closest papers by meaning

• On the Surprising Effectiveness of Attention Transfer for Vision Transformers (NeurIPS 2024 · similarity 0.94)
• No Head Left Behind – Multi-Head Alignment Distillation for Transformers (AAAI 2024 · similarity 0.88)
• ScaleKD: Strong Vision Transformers Could Be Excellent Teachers (NeurIPS 2024 · similarity 0.88)
• Vision Transformers with Self-Distilled Registers (NeurIPS 2025 · similarity 0.88)
• You Only Need Less Attention at Each Stage in Vision Transformers (CVPR 2024 · similarity 0.87)
• Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers (Student Abstract) (AAAI 2024 · similarity 0.87)