Research map / Architectures and efficiency / Attention and transformers
MetaFormer Is Actually What You Need for Vision
Check your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the MetaFormer architecture with pooling as token mixer | PoolFormer · ImageNet-1K validation | top-1 accuracy (%): 82.1 vs 81.8; 82.1 vs 81.0 | Helped |
| replace pooling with identity mapping (ablation) | PoolFormer-S12 · ImageNet-1K validation | top-1 accuracy (%): 77.2 -> 74.3 | Hurt |
| replace pooling with a frozen global random matrix (ablation) | PoolFormer-S12 · ImageNet-1K validation | top-1 accuracy (%): 77.2 -> 75.8 | Hurt |
| replace pooling with depthwise convolution (ablation) | PoolFormer-S12 · ImageNet-1K validation | top-1 accuracy (%): 77.2 -> 78.1 | Helped |
| increase pooling size from 3 to 5 (ablation) | PoolFormer-S12 · ImageNet-1K validation | top-1 accuracy (%): 77.2 -> 77.2 | No clear effect |
| increase pooling size from 3 to 7 (ablation) | PoolFormer-S12 · ImageNet-1K validation | top-1 accuracy (%): 77.2 -> 77.1 | No clear effect |
| increase pooling size from 3 to 9 (ablation) | PoolFormer-S12 · ImageNet-1K validation | top-1 accuracy (%): 77.2 -> 76.8 | Hurt |
| use modified LayerNorm instead of LayerNorm or BatchNorm (ablation) | PoolFormer-S12 · ImageNet-1K validation | top-1 accuracy (%): 76.5 -> 77.2; 76.4 -> 77.2 | Helped |
| remove normalization (ablation) | PoolFormer-S12 · ImageNet-1K validation | top-1 accuracy (%): 77.2 -> 46.1 | Hurt |
| replace GELU with ReLU (ablation) | PoolFormer-S12 · ImageNet-1K validation | top-1 accuracy (%): 77.2 -> 76.4 | Hurt |
| replace GELU with SiLU (ablation) | PoolFormer-S12 · ImageNet-1K validation | top-1 accuracy (%): 77.2 -> 77.2 | No clear effect |
| remove the residual connection (ablation) | PoolFormer-S12 · ImageNet-1K validation | top-1 accuracy (%): 77.2 -> 0.1 | Hurt |
Closest papers by meaning
- Pool Me Wisely: On the Effect of Pooling in Transformer-Based Models (NeurIPS 2025 · similarity 0.87)
- An Image is Worth More Than 16x16 Patches: Exploring Transformers on Individual Pixels (ICLR 2025 · similarity 0.87)
- SHViT: Single-Head Vision Transformer with Memory Efficient Macro Design (CVPR 2024 · similarity 0.86)
- One Meta-tuned Transformer is What You Need for Few-shot Learning (ICML 2024 · similarity 0.86)
- Frequency-Aware Token Reduction for Efficient Vision Transformer (NeurIPS 2025 · similarity 0.85)
- Vision Transformers with Self-Distilled Registers (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.