Research map / Architectures and efficiency / CNN design
Structured Initialization for Vision Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes initializing ViT attention maps to random impulse convolution structures via an SVD-based solve for Q and K, improving small/medium-scale training without architectural change.
Problem
ViTs underperform CNNs on small-scale datasets due to weak inductive bias; existing initialization methods (mimetic, weight sampling from pretrained models) rely on pretrained weights or don't impose structure on attention maps.
Approach
Motivated by the finding that random impulse convolution filters match trained ConvMixer filters, the method initializes each head's softmax attention map to an impulse convolution pattern by solving for Q and K via SVD using positional encodings as pseudo-input, leaving the architecture and training unchanged.
Key results
- On small/medium datasets, impulse init improves ViT-Tiny over default init by about 2-5% on medium datasets and up to 24% on Pets (50.84 vs 26.58).
- On ImageNet-1K, impulse init matches default: ViT-Base 81.83 vs 81.24 default, ViT-Tiny 72.76 vs 72.71.
- Swin-T on CIFAR-10 improves from 89.85 to 91.19 and MLP-Mixer on CIFAR-10 from 87.00 to 88.78 with impulse init.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace default initialization with impulse initialization | ViT-Tiny · Food-101 | accuracy: 77.95 -> 81.85 | Helped |
| Replace default initialization with impulse initialization | ViT-Tiny · CIFAR-10 | accuracy: 92.29 -> 94.67 | Helped |
| Replace default initialization with impulse initialization | ViT-Tiny · CIFAR-100 | accuracy: 71.67 -> 77.02 | Helped |
| Replace default initialization with impulse initialization | ViT-Tiny · STL-10 | accuracy: 61.86 -> 70.21 | Helped |
| Replace default initialization with impulse initialization | ViT-Tiny · Flowers | accuracy: 64.60 -> 73.18 | Helped |
| Replace default initialization with impulse initialization | ViT-Tiny · Pets | accuracy: 26.58 -> 50.84 | Helped |
| Replace default initialization with impulse initialization | ViT-Base · ImageNet-1K | accuracy: 81.24 -> 81.83 | Helped |
| Initialize relative positional embeddings with impulse patterns | Swin-T · CIFAR-10 | accuracy: 89.85 -> 91.19 | Helped |
| Initialize spatial MLPs with a convolutional structure | MLP-Mixer · CIFAR-10 | accuracy: 87.00 -> 88.78 | Helped |
| Use impulse rather than box spatial filters (ablation) | ConvMixer, embedding dimension 256, filter size 3 · CIFAR-10 | accuracy: 81.70 -> 90.68 | Helped |
| Use impulse rather than trained spatial filters (ablation) | ConvMixer, embedding dimension 256, filter size 3 · CIFAR-10 | accuracy: 91.76 -> 90.68 | Hurt |
| Increase embedding dimension from 256 to 512 with impulse filters (ablation) | ConvMixer, filter size 3 · CIFAR-10 | accuracy: 90.68 -> 92.20 | Helped |
Surprising findings
- Mimetic initialization degrades ViT-Base performance relative to default in the final training stage on ImageNet-1K, and falls below default for ViT-Base (80.56 vs 81.24).
Limitations
- Positional encoding is used as pseudo-input, and its influence on attention may diminish with depth.
- Limited number of heads in ViTs may be insufficient to span the filter space of small kernels, so the theoretical guarantee is weaker for ViT.
- Value and projection matrices are not initialized by the method.
Experimental setup
Datasets: Food-101 CIFAR-10 CIFAR-100 STL-10 Flowers Pets ImageNet-1K ImageNet-21K
Benchmarks: Food-101 CIFAR-10 CIFAR-100 STL-10 Flowers Pets ImageNet-1K
Metrics: top-1 classification accuracy
Baselines: default ViT initialization mimetic initialization weight selection
Builds on: mimetic initialization weight selection ConvMixer DeiT training recipe
Closest papers by meaning
- Early Convolutions Help Transformers See Better (NeurIPS 2021 · similarity 0.89)
- Bootstrapping ViTs: Towards Liberating Vision Transformers From Pre-Training (CVPR 2022 · similarity 0.89)
- Slicing Vision Transformer for Flexible Inference (NeurIPS 2024 · similarity 0.88)
- You Only Need Less Attention at Each Stage in Vision Transformers (CVPR 2024 · similarity 0.87)
- Vision Transformer Neural Architecture Search for Out-of-Distribution Generalization: Benchmark and Insights (NeurIPS 2024 · similarity 0.87)
- Bi-ViT: Pushing the Limit of Vision Transformer Quantization (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.