Research map / Architectures and efficiency / Attention and transformers
BiFormer: Vision Transformer With Bi-Level Routing Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use bi-level routing attention instead of the compared sparse attention patterns | Swin-T layout · ImageNet-1K | Top-1 accuracy (%): 81.4 -> 82.7 | Helped |
| use bi-level routing attention instead of the compared sparse attention patterns | Swin-T layout · ADE20K | mIoU (%): 41.5 -> 44.8 | Helped |
| replace non-overlapped patch embedding with overlapped patch embedding (ablation) | Swin-T layout · ImageNet-1K | Top-1 accuracy (%): 82.7 -> 82.8 | Helped |
| use a deeper layout, with fewer base channels and a smaller MLP expansion ratio (ablation) | overlapped-patch Swin-T layout · ImageNet-1K | Top-1 accuracy (%): 82.8 -> 83.5 | Helped |
| add convolution position encoding (ablation) | deeper Swin-T layout · ImageNet-1K | Top-1 accuracy (%): 83.5 -> 83.8 | Helped |
| add token labeling training (ablation) | BiFormer-S with convolution position encoding · ImageNet-1K | Top-1 accuracy (%): 83.8 -> 84.3 | Helped |
| reduce top-k from 1,4,16,49 to 1,2,8,32 (ablation) | BiFormer-STL · ImageNet-1K | Top-1 accuracy: 82.7 -> 82.4 | Hurt |
| increase top-k from 1,4,16,49 to 2,8,32,49 (ablation) | BiFormer-STL · ImageNet-1K | Top-1 accuracy: 82.7 -> 82.6 | Hurt |
| replace local window attention with BRA (ablation) | pretrained DeiT-B adapted for dense prediction · ADE20K | mIoU (%): 43.55 -> 45.92 | Helped |
| use BRA plus 4 global propagation blocks instead of local window attention plus 4 global propagation blocks (ablation) | pretrained DeiT-B adapted for dense prediction · ADE20K | mIoU (%): 46.64 -> 46.84 | Helped |
| use sparse sampling rather than downsampling | BiFormer on COCO object detection · COCO 2017, RetinaNet | AP_S: 30.2 vs 29.2 | Helped |
| use query-adaptive region routing and sparse token attention | BiFormer · COCO 2017, Mask R-CNN | box mAP: 47.8 vs 46.7 | Helped |
Closest papers by meaning
- Learned Queries for Efficient Local Attention (CVPR 2022 · similarity 0.88)
- Perceiving Longer Sequences With Bi-Directional Cross-Attention Transformers (NeurIPS 2024 · similarity 0.88)
- BinaryAttention: One-Bit QK-Attention for Vision and Diffusion Transformers (CVPR 2026 · similarity 0.88)
- Bridging the Divide: Reconsidering Softmax and Linear Attention (NeurIPS 2024 · similarity 0.87)
- Token Statistics Transformer: Linear-Time Attention via Variational Rate Reduction (ICLR 2025 · similarity 0.87)
- Bi-ViT: Pushing the Limit of Vision Transformer Quantization (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.