Research map / Detection and segmentation / CNN features
Polyline Path Masked Attention for Vision Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Polyline Path Masked Attention (PPMA), which turns Mamba2's 1D structured mask into a 2D polyline-path mask and injects it into ViT self-attention and criss-cross attention as an explicit positional encoding.
Problem
ViTs' positional encodings do not explicitly model spatial adjacency, and Mamba-style 1D scanning breaks 2D spatial adjacency, so both lag behind Transformer and SSM backbones on high-level vision tasks.
Approach
Define an L-shaped (V2H and H2V) polyline scanning path on the 2D token grid and a corresponding decay mask whose product of horizontal and vertical input-dependent decays decomposes into two 1D structured masks, enabling efficient O(N^1.5) multiplication; the mask is Hadamard-multiplied into softmax attention of ViTs.
Key results
- PPMA-S reaches 84.2% top-1 on ImageNet-1K, versus 84.0% for RMT-S, with similar FLOPs (4.9G).
- PPMA-B reaches 52.3% SS mIoU on ADE20K (vs. 52.0% for RMT-B) and 51.1 box AP on COCO Mask R-CNN 1x (vs. 50.7 for RMT-B).
- In the ablation on PPMA-T, the 2D polyline path mask adds 0.32% top-1 on ImageNet-1K and 0.95% SS mIoU on ADE20K over the no-mask baseline, but lowers throughput from 1779 to 1034 imgs/s.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the 2D polyline path mask (ablation) | PPMA-T · ImageNet-1K | Top-1 accuracy: +0.32% | Helped |
| add the 2D polyline path mask (ablation) | PPMA-T · ADE20K | SS mIoU: +0.95% | Helped |
| replace no mask with the RMT decay mask (ablation) | PPMA-T · ImageNet-1K | Top-1 accuracy: 82.28 -> 82.35 | Helped |
| replace no mask with the RMT decay mask (ablation) | PPMA-T · ADE20K | SS mIoU: 47.78 -> 48.01 | Helped |
| add the cross scan mask (ablation) | PPMA-T · ImageNet-1K | Top-1 accuracy: 82.28 -> 82.44 | Helped |
| add the cross scan mask (ablation) | PPMA-T · ADE20K | SS mIoU: 47.78 -> 48.14 | Helped |
| add the Hilbert scan mask (ablation) | PPMA-T · ImageNet-1K | Top-1 accuracy: 82.28 -> 82.44 | Helped |
| add the Hilbert scan mask (ablation) | PPMA-T · ADE20K | SS mIoU: 47.78 -> 48.14 | Helped |
| use a V2H polyline path mask (ablation) | PPMA-T · ImageNet-1K | Top-1 accuracy: 82.28 -> 82.44 | Helped |
| use a V2H polyline path mask (ablation) | PPMA-T · ADE20K | SS mIoU: 47.78 -> 48.57 | Helped |
| use separate horizontal and vertical decay factors instead of a shared factor (ablation) | PPMA-T · ImageNet-1K | Top-1 accuracy: 82.37 -> 82.60 | Helped |
| use separate horizontal and vertical decay factors instead of a shared factor (ablation) | PPMA-T · ADE20K | SS mIoU: 48.27 -> 48.73 | Helped |
Surprising findings
- The polyline path mask lowers throughput substantially (1779 to 1034 imgs/s), and the authors state this overhead as a limitation.
- The RMT decay mask gives only a small gain over no mask (82.35 vs 82.28 top-1), while the proposed mask gives larger gains on segmentation.
Limitations
- Adds GPU memory and reduces throughput compared to baseline attention.
- Efficient CUDA or Triton kernels are not yet implemented.
- Gains over RMT are small on ImageNet-1K (e.g., +0.2% for S, +0.2% for T).
Experimental setup
Datasets: ImageNet-1K MSCOCO2017 ADE20K
Benchmarks: ImageNet-1K MSCOCO2017 ADE20K
Metrics: top-1 accuracy box AP mask AP mIoU FLOPs throughput
Baselines: RMT VMamba MLLA 2DMamba Spatial-Mamba MambaVision Swin CSWin ConvNeXt GrootVL
Builds on: Mamba2 State Space Duality RMT criss-cross attention UPerNet
Closest papers by meaning
- MambaVision: A Hybrid Mamba-Transformer Vision Backbone (CVPR 2025 · similarity 0.85)
- Demystify Mamba in Vision: A Linear Attention Perspective (NeurIPS 2024 · similarity 0.85)
- Multi-Scale VMamba: Hierarchy in Hierarchy Visual State Space Model (NeurIPS 2024 · similarity 0.85)
- One-dimensional Path Convolution (ICML 2025 · similarity 0.84)
- RMT: Retentive Networks Meet Vision Transformers (CVPR 2024 · similarity 0.84)
- You Only Need Less Attention at Each Stage in Vision Transformers (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.