Research map / Architectures and efficiency / Attention and transformers
CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddings
Tatiana Likhomanenko, Qiantong Xu, Gabriel Synnaeve, Ronan Collobert, Alex Rogozhnikov
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace sinpos with CAPE | ViT-B trained at 224^2 · ImageNet val, eval 672^2 | ImageNet val top-1: 70.714 -> 73.426 | Improved |
| replace abspos with CAPE | ViT-B trained at 224^2 · ImageNet val, eval 672^2 | ImageNet val top-1: 72.206 -> 73.426 | Improved |
| replace nopos with CAPE | ViT-B trained at 224^2 · ImageNet val, eval 160^2 | ImageNet val top-1: 75.296 -> 77.138 | Improved |
| train on mixture of resolutions (UniViT) instead of 224^2 only, with CAPE lambda=1 | ViT-B CAPE · ImageNet val, eval 672^2 | ImageNet val top-1: 73.064 -> 82.34 | Improved |
| replace sinpos with CAPE lambda=1 in UniViT | UniViT · ImageNet val, eval 224^2 | ImageNet val top-1: 80.824 -> 81.26 | Improved |
| choose optimal resizing per image-size bin at inference | UniViT CAPE · ImageNet val | ImageNet val top-1: 82.55 -> 82.92 | Improved |
| test time resolution augmentation (average logits over r, r-32, r+32) | sinpos/CAPE ViT and UniViT · ImageNet val | ImageNet val top-1: 0.2%-0.5% | Improved |
| rescale positions to [-r/224, r/224] instead of [-1,1] at low resolution (ablation) | sinpos ViT · ImageNet val, eval 160^2 | ImageNet val top-1: 77.15 -> 77.89 | Improved |
| rescale positions to [-r/224, r/224] instead of [-1,1] at high resolution (ablation) | sinpos ViT · ImageNet val, eval 384^2 | ImageNet val top-1: 79.72 -> 72.01 | Hurt |
| remove local shift from CAPE (epsilon=0) (ablation) | CAPE ViT · ImageNet val, eval 672^2 | ImageNet val top-1: 73.426 -> 73.354 | No clear effect |
| remove global scaling (lambda=1) from UniViT CAPE (ablation) | UniViT CAPE · ImageNet val, eval 224^2 | ImageNet val top-1: 81.164 -> 81.26 | No clear effect |
| add CAPE to sinpos baseline (no SpecAugment) (ablation) | TED-LIUM v3 acoustic model · TED-LIUM v3 validation | WER: 4-5% | Improved |
Closest papers by meaning
- Functional Interpolation for Relative Positions improves Long Context Transformers (ICLR 2024 · similarity 0.86)
- Causality-Induced Positional Encoding for Transformer-Based Representation Learning of Non-Sequential Features (NeurIPS 2025 · similarity 0.86)
- DAPE: Data-Adaptive Positional Encoding for Length Extrapolation (NeurIPS 2024 · similarity 0.86)
- What Rotary Position Embedding Can Tell Us: Identifying Query and Key Weights Corresponding to Basic Syntactic or High-level Semantic Information (NeurIPS 2024 · similarity 0.84)
- StableMask: Refining Causal Masking in Decoder-only Transformer (ICML 2024 · similarity 0.84)
- Algebraic Positional Encodings (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
