AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddings

Tatiana Likhomanenko, Qiantong Xu, Gabriel Synnaeve, Ronan Collobert, Alex Rogozhnikov

NeurIPS 2021 · Architectures and efficiency › Attention and transformers · approach: Transformer expressivity and learnability · 7 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
replace sinpos with CAPEViT-B trained at 224^2 · ImageNet val, eval 672^2ImageNet val top-1: 70.714 -> 73.426Improved
replace abspos with CAPEViT-B trained at 224^2 · ImageNet val, eval 672^2ImageNet val top-1: 72.206 -> 73.426Improved
replace nopos with CAPEViT-B trained at 224^2 · ImageNet val, eval 160^2ImageNet val top-1: 75.296 -> 77.138Improved
train on mixture of resolutions (UniViT) instead of 224^2 only, with CAPE lambda=1ViT-B CAPE · ImageNet val, eval 672^2ImageNet val top-1: 73.064 -> 82.34Improved
replace sinpos with CAPE lambda=1 in UniViTUniViT · ImageNet val, eval 224^2ImageNet val top-1: 80.824 -> 81.26Improved
choose optimal resizing per image-size bin at inferenceUniViT CAPE · ImageNet valImageNet val top-1: 82.55 -> 82.92Improved
test time resolution augmentation (average logits over r, r-32, r+32)sinpos/CAPE ViT and UniViT · ImageNet valImageNet val top-1: 0.2%-0.5%Improved
rescale positions to [-r/224, r/224] instead of [-1,1] at low resolution (ablation)sinpos ViT · ImageNet val, eval 160^2ImageNet val top-1: 77.15 -> 77.89Improved
rescale positions to [-r/224, r/224] instead of [-1,1] at high resolution (ablation)sinpos ViT · ImageNet val, eval 384^2ImageNet val top-1: 79.72 -> 72.01Hurt
remove local shift from CAPE (epsilon=0) (ablation)CAPE ViT · ImageNet val, eval 672^2ImageNet val top-1: 73.426 -> 73.354No clear effect
remove global scaling (lambda=1) from UniViT CAPE (ablation)UniViT CAPE · ImageNet val, eval 224^2ImageNet val top-1: 81.164 -> 81.26No clear effect
add CAPE to sinpos baseline (no SpecAugment) (ablation)TED-LIUM v3 acoustic model · TED-LIUM v3 validationWER: 4-5%Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy