Research map / Architectures and efficiency / Attention and transformers
Maximizing the Position Embedding for Vision Transformers with Global Average Pooling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MPVG, a layer-wise position-embedding scheme for vision transformers with global average pooling that adds PE to the final LayerNorm to keep PE's counterbalancing role, improving ImageNet, CIFAR-100, COCO and ADE20K results.
Problem
Layer-wise position embedding delivery, which helps ViTs, conflicts with global average pooling: combining the two lowers accuracy compared with either alone.
Approach
Observes that in layer-wise PE, PE counterbalances token-embedding values across layers, and that this counterbalancing is lost after the last layer; MPVG re-adds PE to the final LayerNorm (Last LN) and excludes PE delivery to layer 0, with PE added to tokens before layer 0.
Key results
- On ImageNet-1K, MPVG raises DeiT-Ti top-1 from 72.14% to 73.51% and DeiT-S from 79.81% to 80.61%, outperforming LaPE and PVG across all listed models.
- On CIFAR-100, MPVG improves ViT-Lite from 74.90% to 76.87%.
- On ADE20K with ViT-Adapter-Ti, MPVG reaches 41.69 mIoU versus 40.55 default; on COCO, 46.5 box AP versus 45.9 default.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add initial PE to the Last LN in PVG (MPVG) | DeiT-Ti · ImageNet-1K | Top-1 accuracy (%): 73.17 -> 73.51 | Helped |
| Add initial PE to the Last LN in PVG (MPVG) | DeiT-S, DeiT-B, Swin-Ti, CeiT-Ti, and T2T-ViT-7 · ImageNet-1K | Top-1 accuracy (%): 80.38 -> 80.61; 82.21 -> 82.42; 81.52 -> 81.64; 77.14 -> 77.20; 71.91 -> 72.28 | Helped |
| Add initial PE to the Last LN in PVG (MPVG) | ViT-Lite and T2T-ViT-7 · CIFAR-100 | Top-1 accuracy (%): 76.67 -> 76.87; 83.39 -> 83.51 | Helped |
| Add initial PE to the Last LN in PVG (MPVG) | ViT-Adapter-Ti · COCO 2017 | Box AP: +0.5 | Helped |
| Add initial PE to the Last LN in PVG (MPVG) | ViT-Adapter-Ti · COCO 2017 | Mask AP: +0.4 | Helped |
| Add initial PE to the Last LN in PVG (MPVG) | ViT-Adapter-Ti · ADE20K | mIoU: +0.62 | Helped |
| Use pos_11 rather than pos_0 in the Last LN (ablation) | MPVG on DeiT-Ti · ImageNet-1K | Top-1 accuracy (%): 73.51 -> 73.30 | Hurt |
| Use pos_8 rather than pos_0 in the Last LN (ablation) | MPVG on DeiT-Ti · ImageNet-1K | Top-1 accuracy (%): 73.51 -> 73.38 | Hurt |
| Use pos_5 rather than pos_0 in the Last LN (ablation) | MPVG on DeiT-Ti · ImageNet-1K | Top-1 accuracy (%): 73.51 -> 73.39 | Hurt |
| Exclude layer 0 when delivering PE (ablation) | MPVG on DeiT-Ti · ImageNet-1K | Top-1 accuracy (%): 73.31 -> 73.51 | Helped |
| Use hierarchical rather than non-hierarchical PE delivery (ablation) | MPVG on DeiT-Ti · ImageNet-1K | Top-1 accuracy (%): 73.48 -> 73.51 | No clear effect |
| Add PE to token embedding before layer 0 (ablation) | MPVG on DeiT-Ti · ImageNet-1K | Top-1 accuracy (%): 73.28 -> 73.51 | Helped |
Surprising findings
- Adding PE to the Last LN without layer-wise delivery decreases accuracy (72.14% vs 72.40% with GAP only), so the benefit depends on the layer-wise structure.
- Delivering PE to layer 0 is unnecessary and reduces performance when excluded in the proposed structure.
Limitations
- MPVG is incompatible with the class token method.
- Results are reported on relatively small ViT variants (Ti, S, B) and a limited set of architectures.
- The counterbalancing explanation is based on observational heatmaps and correlation analysis rather than a formal theory.
Experimental setup
Datasets: ImageNet-1K CIFAR-100 COCO 2017 ADE20K
Benchmarks: ImageNet-1K CIFAR-100 COCO 2017 ADE20K
Metrics: Top-1 accuracy box AP mask AP mIoU
Baselines: Default (absolute PE) LaPE PVG
Builds on: ViT LaPE DeiT ViT-Adapter
Closest papers by meaning
- Circle-RoPE: Cone-like Decoupled Rotary Positional Embedding for Vision-Language Models (ICML 2026 · similarity 0.86)
- Exploring Token Pruning in Vision State Space Models (NeurIPS 2024 · similarity 0.85)
- Parabolic Position Encoding: Vision-Centric, Principled, Extrapolatable, General (NeurIPS 2026 · similarity 0.85)
- Vision Transformers with Self-Distilled Registers (NeurIPS 2025 · similarity 0.85)
- Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization (NeurIPS 2025 · similarity 0.85)
- ComRoPE: Scalable and Robust Rotary Position Embedding Parameterized by Trainable Commuting Angle Matrices (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.