Research map / Architectures and efficiency / Attention and transformers

PolaFormer: Polarity-aware Linear Attention for Vision Transformers

ICLR 2025 · Architectures and efficiency › Attention and transformers · approach: Attention head and block redesign · 6 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PolaFormer, a linear attention for vision transformers that splits query-key interactions by polarity (same- vs opposite-signed) and rescales with a learnable power function to lower attention entropy.

Problem

Linear attention approximations of softmax lose negative query-key interactions (due to non-negative feature maps) and produce high-entropy, uniform attention maps, reducing expressiveness on vision tasks.

Approach

Decompose q and k into positive and negative parts, compute same-signed and opposite-signed interactions in two value half-streams with learnable sign-aware coefficient matrices G^s and G^o, and apply a learnable channel-wise power function (exponent >1) that provably reduces positive sequence entropy.

Key results

• DeiT-T-PolaFormer reaches 74.6% top-1 on ImageNet-1K, +2.4 over DeiT-T (72.2%), and Swin-T-PolaFormer reaches 82.6% vs 81.2% for Swin-T.
• PVT-T-PolaFormer gains +3.7 top-1 on ImageNet and +4.6 AP^b_75 on COCO with Mask R-CNN over PVT-T baseline.
• PolaFormer_alpha=3 achieves 60.72% average on Long Range Arena, the highest among listed linear attention models (Hedgehog 59.66%).

Measured effects reported in the paper

Surprising findings

• Replacing ReLU-style feature maps with the learnable power function does not uniformly help; the paper reports results are relatively insensitive to alpha (within 2%) on LRA classification.
• Detection gains (up to +4.6 AP) are larger than classification gains, which the authors attribute to needs for fine-grained attention maps.

Limitations

• Gains on classification with larger models are small (e.g., Swin-B +0.3).
• Requires depthwise convolution to resolve low-rank degeneracy; the rank analysis is motivational rather than fully quantified.
• Theorem uses an independence assumption across dimensions and only shows existence of an entropy-reducing function class, with power function chosen for simplicity.

Experimental setup

Datasets: ImageNet-1K COCO ADE20K Long Range Arena

Benchmarks: ImageNet-1K COCO ADE20K Long Range Arena

Metrics: Top-1 accuracy box AP mask AP mIoU accuracy throughput peak memory

Baselines: DeiT Swin Transformer PVT PVTv2 FLatten Transformer Hydra Attention Efficient Attention Skyformer Hedgehog Performer Cosformer

Builds on: linear attention FLatten Transformer Skyformer Swin Transformer PVT

Closest papers by meaning

• Bridging the Divide: Reconsidering Softmax and Linear Attention (NeurIPS 2024 · similarity 0.88)
• ZeroS: Zero‑Sum Linear Attention for Efficient Transformers (NeurIPS 2025 · similarity 0.87)
• Degrees of Freedom for Linear Attention: Distilling Softmax Attention with Optimal Feature Efficiency (NeurIPS 2025 · similarity 0.86)
• Breaking the Low-Rank Dilemma of Linear Attention (CVPR 2025 · similarity 0.86)
• MLP Can Be A Good Transformer Learner (CVPR 2024 · similarity 0.86)
• Mixture-of-Top-$k$ Attention: Efficient Attention as Scalable Fast Weights (NeurIPS 2026 · similarity 0.85)