Research map / Architectures and efficiency / Attention and transformers
Mimetic Initialization of Self-Attention Layers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use mimetic self-attention initialization instead of base initialization | ViT-Tiny · CIFAR-10, width 192, depth 12, 3 heads | accuracy: 86.07 -> 90.78 | Helped |
| Use mimetic self-attention initialization instead of base initialization | ViT-Ti · ImageNet-1k, ResNet-style pipeline, 150 epochs, batch size 1024 | accuracy: 67.80 -> 71.92 | Helped |
| Use mimetic self-attention initialization instead of base initialization | ViT-Ti · ImageNet-1k, DeiT-style pipeline, 300 epochs | accuracy: 72.08 -> 72.65 | Helped |
| Use mimetic self-attention initialization instead of base initialization | ViT-Tiny · CIFAR-100 | accuracy: 60.94 -> 67.33 | Helped |
| Use mimetic self-attention initialization instead of base initialization | Transformer · WikiText-103 | perplexity: 28.87 -> 28.21 | Helped |
| Use random instead of sinusoidal position embeddings with mimetic initialization (ablation) | ViT-Ti · CIFAR-10 | accuracy: 91.38 -> 88.70 | Hurt |
| Remove mimetic attention initialization, keeping sinusoidal position embeddings (ablation) | ViT-Ti · CIFAR-10 | accuracy: 91.38 -> 87.39 | Hurt |
| Initialize only query and key weights, omitting value and projection initialization (ablation) | ViT-Ti · CIFAR-10 | accuracy: 91.38 -> 89.17 | Hurt |
| Initialize only value and projection weights, omitting query and key initialization (ablation) | ViT-Ti · CIFAR-10 | accuracy: 91.38 -> 87.23 | Hurt |
| Set the value-projection product diagonal positive instead of negative (ablation) | ViT-Ti · CIFAR-10 | accuracy: 91.38 -> 89.65 | Hurt |
| Add a negative value-projection diagonal initialization to GPSA (ablation) | GPSA, ViT-Ti · CIFAR-10 | accuracy: 90.83 -> 91.21 | Helped |
| Replace mimetic value-projection initialization with original LayerScale (ablation) | ViT-Ti · CIFAR-10 | accuracy: 91.38 -> 89.90 | Hurt |
Closest papers by meaning
- Structured Initialization for Vision Transformers (NeurIPS 2025 · similarity 0.88)
- In-context Convergence of Transformers (ICML 2024 · similarity 0.86)
- One Meta-tuned Transformer is What You Need for Few-shot Learning (ICML 2024 · similarity 0.85)
- On the Role of Attention Masks and LayerNorm in Transformers (NeurIPS 2024 · similarity 0.85)
- A Theoretical Analysis of Self-Supervised Learning for Vision Transformers (ICLR 2025 · similarity 0.85)
- Sinusoidal Initialization, Time for a New Start (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.