Research map / Architectures and efficiency / Attention and transformers
Conditional Positional Encodings for Vision Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove positional encodings entirely from DeiT-tiny (ablation) | DeiT-tiny · ImageNet val @224 | top-1 accuracy: 72.2 -> 68.2 | Hurt |
| replace learnable absolute PE with conditional PEG (depth-wise 3x3 conv at position 0-5) | DeiT-tiny · ImageNet val @224 | top-1 accuracy: 72.2 -> 73.4 | Helped |
| PEG-based conditional PE evaluated at higher resolution without fine-tuning | CPVT-Ti vs DeiT-tiny · ImageNet val @384 | top-1 accuracy: +3.0 over DeiT-tiny (74.2 vs 71.2) | Helped |
| replace class token with global average pooling head (ablation) | CPVT-Ti · ImageNet val @224 | top-1 accuracy: 73.4 -> 74.9 | Helped |
| replace class token with global average pooling head (ablation) | DeiT-tiny · ImageNet val @224 | top-1 accuracy: 72.2 -> 72.6 | No clear effect |
| replace PEG with 12 kernel-1 learnable convolutions (more parameters, no local neighborhood) (ablation) | DeiT-tiny · ImageNet val @224 | top-1 accuracy: 72.4 -> 68.6 (vs PEG) | Hurt |
| PEG with frozen random 3x3 weights (no learning) (ablation) | DeiT-tiny · ImageNet val @224 | top-1 accuracy: 68.2 -> 71.3 (vs no PE) | Helped |
| apply PEG only to Q and K rather than Q, K and V (ablation) | DeiT-tiny with PEG · ImageNet val @224 | top-1 accuracy: 71.3 vs 72.4 | No clear effect |
| remove zero paddings from PEG (ablation) | CPVT-S · ImageNet val @224 | top-1 accuracy: 72.4 -> 70.5 (tiny) | Hurt |
| place PEG at position -1 (before first encoder) vs position 0 (ablation) | DeiT-tiny with PEG · ImageNet val @224 | top-1 accuracy: 72.4 -> 70.6 | Hurt |
| enlarge PEG kernel to 27x27 at position -1 (ablation) | DeiT-tiny with PEG at position -1 · ImageNet val @224 | top-1 accuracy: 70.6 -> 72.5 | Helped |
| add PEG to learnable absolute PE (single layer) (ablation) | DeiT-tiny · ImageNet val @224 | top-1 accuracy: 72.2 -> 72.9 | Helped |
Closest papers by meaning
- Cameras as Relative Positional Encoding (NeurIPS 2025 · similarity 0.88)
- DPPE: Rethinking Camera-Based Positional Encoding for Scaling Multi-View Transformers (NeurIPS 2026 · similarity 0.87)
- Maximizing the Position Embedding for Vision Transformers with Global Average Pooling (AAAI 2025 · similarity 0.86)
- Parabolic Position Encoding: Vision-Centric, Principled, Extrapolatable, General (NeurIPS 2026 · similarity 0.86)
- GTA: A Geometry-Aware Attention Mechanism for Multi-View Transformers (ICLR 2024 · similarity 0.86)
- LookHere: Vision Transformers with Directed Attention Generalize and Extrapolate (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.