Research map / Architectures and efficiency / Attention and transformers
Star with Bilinear Mapping
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Star with Bilinear Mapping (SBM), a hierarchical vision backbone that combines element-wise star operations with a low-rank bilinear mapping branch to get global context at linear complexity, evaluated on ImageNet-1K, CIFAR-100, ADE20K and COCO.
Problem
Quadratic complexity of full self-attention in Transformers limits scaling to high-resolution vision inputs, while star operations (element-wise multiplication) lack a global receptive field.
Approach
Augments the star operation with a bilinear mapping branch whose spatial projection is low-rank decomposed, giving global token interactions at linear complexity, plus a multiplicative mapping-control gate G(x) and a linear branch V, all fused by star operations in a Swin-like four-stage backbone.
Key results
- SBM-T reaches 83.2% top-1 on ImageNet-1K with 25M params and 4.5G FLOPs, above VMamba-T (82.6%) and Swin-T (81.3%).
- SBM-B reaches 85.1% top-1 on ImageNet-1K with 93M params, and 50.2 mIoU on ADE20K with UperNet.
- On COCO Mask R-CNN 1x, SBM-B gets 50.2 APb and 44.8 APm, outperforming VMamba-B by 1.7 points on both.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SBM-T instead of MambaOut-T | ImageNet-1K image classification · ImageNet-1K | Top-1 accuracy: 82.7 -> 83.2 | Helped |
| use SBM-S instead of MambaOut-S | ImageNet-1K image classification · ImageNet-1K | Top-1 accuracy: 84.1 -> 84.4 | Helped |
| use SBM-B instead of MambaOut-B | ImageNet-1K image classification · ImageNet-1K | Top-1 accuracy: 84.2 -> 85.1 | Helped |
| use SBM-B instead of DeiT-B | CIFAR-100 fine-tuning · CIFAR-100 fine-tuning | Top-1 accuracy: 90.5 -> 91.9 | Helped |
| use SBM-B instead of DeiT-B | CIFAR-100 linear probing · CIFAR-100 linear probing | Top-1 accuracy: 80.6 -> 81.7 | Helped |
| remove linear mapping branch (ablation) | SBM-T · ImageNet-1K | Top-1 accuracy: 83.2 -> 82.3 | Hurt |
| remove bilinear mapping (ablation) | SBM-T · ImageNet-1K | Top-1 accuracy: 83.2 -> 81.6 | Hurt |
| remove mapping control branch (ablation) | SBM-T · ImageNet-1K | Top-1 accuracy: 83.2 -> 82.6 | Hurt |
| add absolute positional encoding (ablation) | SBM-T baseline block · ImageNet-1K | Top-1 accuracy: 82.3 -> 82.6 | Helped |
| add conditional positional encoding (ablation) | SBM-T baseline block · ImageNet-1K | Top-1 accuracy: 82.3 -> 82.9 | Helped |
| add shortcut connections (ablation) | SBM-T baseline block · ImageNet-1K | Top-1 accuracy: 82.3 -> 82.4 | Helped |
| add layer normalization (ablation) | SBM-T baseline block · ImageNet-1K | Top-1 accuracy: 73.4 -> 82.3 | Helped |
Surprising findings
- Removing the bilinear mapping branch causes a larger accuracy drop (-1.6) than removing the linear mapping branch (-0.9) in the ablation.
- Removing layer normalization drops top-1 accuracy from 82.3% to 73.4%, showing LN is critical for stable training.
Limitations
- Throughput advantage is partly reduced by the mapping control branch and full block design (SBM-T full block 732 vs 860 baseline).
- Classification and segmentation evaluated only on standard benchmarks; no large-scale pretraining or language/video tasks reported.
- Linear complexity claim relies on a low-rank approximation with dimension m set comparable to d, so the asymptotic advantage depends on that choice.
Experimental setup
Datasets: ImageNet-1K CIFAR-100 ADE20K COCO 2017
Benchmarks: ImageNet-1K CIFAR-100 ADE20K COCO 2017
Metrics: top-1 accuracy mIoU APb APm throughput FLOPs parameters
Baselines: Swin Transformer ConvNeXt VMamba LocalVMamba MambaOut DeiT Focal CSwin NAT Evo-ViT EViT TPS
Builds on: Swin Transformer MambaOut VMamba Rewrite the Star (star operation) bilinear mapping Conditional Positional Encoding
Closest papers by meaning
- Perceiving Longer Sequences With Bi-Directional Cross-Attention Transformers (NeurIPS 2024 · similarity 0.84)
- xT: Nested Tokenization for Larger Context in Large Images (ICML 2024 · similarity 0.83)
- 2DMamba: Efficient State Space Model for Image Representation with Applications on Giga-Pixel Whole Slide Image Classification (CVPR 2025 · similarity 0.83)
- Win-Win: Training High-Resolution Vision Transformers from Two Windows (ICLR 2024 · similarity 0.83)
- Multi-Scale VMamba: Hierarchy in Hierarchy Visual State Space Model (NeurIPS 2024 · similarity 0.82)
- Breaking the Low-Rank Dilemma of Linear Attention (CVPR 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.