Research map / Architectures and efficiency / Attention and transformers

Parabolic Position Encoding: Vision-Centric, Principled, Extrapolatable, General

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Token merging and masked pretraining · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Parabolic Position Encoding (PaPE), a sum-of-learnable-parabolas attention bias over relative positions with context-dependent coefficients, plus a rotation-invariant variant PaPE-RI, evaluated for extrapolation on ImageNet-1K and generality on 8 datasets across 4 vision modalities.

Problem

Position encodings designed for language are only partially adapted to vision; existing vision encodings lack some desirable properties (directionality, distance decay, context awareness, rotation invariance) and extrapolate poorly beyond training resolution.

Approach

Encode relative positions as a context-dependent sum of concave parabolas in the relative position components (distance decay via negative quadratic coefficients, directionality via linear terms, context awareness via token-conditioned coefficients), rewritten as query-key transformations so it works with fused attention kernels; PaPE-RI constrains parameters to be rotation invariant.

Key results

• On ImageNet-1K classification extrapolated to 1024^2 (trained at 224^2), PaPE reaches 74.9% vs 64.4% for next-best LookHere, a 10.5-point gain.
• Across 8 datasets, PaPE achieves the highest average score of 66.3 vs 65.3 for RoPE, matching the best baseline on 5 datasets and exceeding all on UCF101 (49.5 vs 43.9) and GEN1 (34.8 vs 33.6).
• PaPE-RI, the rotation-invariant variant, lags with an average of 63.3 and does not extrapolate as well, suggesting directionality matters for extrapolation.

Surprising findings

• Rotation invariant encodings (nD-ALiBi, PaPE-RI) did not outperform rotation-variant encodings on point cloud tasks.
• Position interpolation hurts all attention-bias encodings including PaPE, while it helps nD-sincos and RoPE.

Limitations

• Moderately increased parameters and compute, scaling with the number m of parabolas.
• Gains on several datasets are small (often within 0.1-1 points) and some datasets show PaPE trailing (ScanNet).
• Position interpolation degrades PaPE, and PaPE-RI does not extrapolate strongly.

Experimental setup

Datasets: ImageNet-1K UCF101 DvsGesture GEN1 COCO ScanNet ModelNet40 nuScenes CIFAR-10

Benchmarks: ImageNet-1K UCF101 DvsGesture GEN1 COCO ScanNet ModelNet40 nuScenes ImageNetV2 ImageNet-Renditions

Metrics: accuracy mAP mIoU

Baselines: nD-sincos RoPE RoPE-Mixed nD-ALiBi LookHere

Builds on: RoPE ALiBi 2D-ALiBi LookHere RoPE-Mixed

Closest papers by meaning

• Revisiting Multimodal Positional Encoding in Vision–Language Models (ICLR 2026 · similarity 0.87)
• DPPE: Rethinking Camera-Based Positional Encoding for Scaling Multi-View Transformers (NeurIPS 2026 · similarity 0.86)
• Cameras as Relative Positional Encoding (NeurIPS 2025 · similarity 0.86)
• Maximizing the Position Embedding for Vision Transformers with Global Average Pooling (AAAI 2025 · similarity 0.85)
• Perception Encoder: The best visual embeddings are not at the output of the network (NeurIPS 2025 · similarity 0.85)
• KeyPoint Relative Position Encoding for Face Recognition (CVPR 2024 · similarity 0.85)