AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Learnable Fourier Features for Multi-dimensional Spatial Positional Encoding

Yang Li, Si Si, Gang Li, Cho-Jui Hsieh, Samy Bengio

NeurIPS 2021 · Architectures and efficiency › Attention and transformers · approach: Attention head and block redesign · 31 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Add MLP on top of learnable Fourier features (ablation)Learnable-Fourier-2/2 · Widget captioningCIDEr: 95.3 -> 98.0Improved
Make Fourier features learnable instead of fixed (ablation)Fourier+MLP-2/2 · Widget captioningCIDEr: 100.2 vs 98.0Mixed
Add MLP to sinusoidal encoding (ablation)Sine-4D · Widget captioningCIDEr: 94.9 -> 98.1Improved
Add Fourier features to MLP-only encoding (ablation)MLP · Widget captioningCIDEr: 70.3 -> 100.7Improved
Add MLP to learnable Fourier features with normal init (ablation)Learnable-Fourier · COCO 2017 valAP: 39.1 -> 40.2Improved
Add MLP to learnable Fourier features with uniform init (ablation)Learnable-Fourier · COCO 2017 valAP: 38.3 -> 40.0Improved
Normal init instead of uniform [0,1] init (ablation)Learnable-Fourier · COCO 2017 valAP: 38.3 -> 39.1Improved
Replace Embed-2D with learnable Fourier+MLPDETR · COCO unseen sizesAP: 36.6 -> 39.5Improved
Replace Sine-2D with learnable Fourier+MLPDETR · COCO 2017 valAP: 40.1 -> 40.2Improved
Replace Embed-4D with Learnable-Fourier+MLP-4/1widget captioning model · unseen positionsCIDEr: 78.5 -> 82.2Improved
Replace Embed-4D with Learnable-Fourier+MLP-4/1widget captioning model · seen positionsCIDEr: 123.4 -> 123.4No clear effect
Group coordinates (2/2 or 4/1) rather than single 1/4 group (ablation)Learnable-Fourier+MLP · Widget captioningCIDEr: 99.1 -> 100.7Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy