Research map / Detection and segmentation / CNN features
Revisiting the Integration of Convolution and Attention for Vision Backbone
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GLMix, a block that runs convolutions on a fine-grained pixel grid and multi-head self-attention on a small set of soft semantic slots, bridged by differentiable soft clustering and dispatching modules, yielding the GLNet backbone family.
Problem
Full-resolution multi-head self-attention alongside convolutions in vision backbones causes quadratic cost and poor scalability with input resolution; the paper asks whether attention is needed at pixel granularity.
Approach
Apply convolutions to a regular feature grid and self-attention to a small set of semantic slots in parallel, linking them with a pair of conjugated, fully differentiable soft clustering (grid to slots) and dispatching (slots to grid) modules, so global attention runs at coarse granularity.
Key results
- GLNet-STL reaches 82.5% ImageNet-1k top-1 versus 81.3% for Swin-T at similar FLOPs (4.4G) and higher throughput (835.9 im/s).
- GLNet-4G/9G/16G reach 83.7%/84.5%/85.0% ImageNet-1k top-1 under standard supervised training; with distillation GLNet-4G reaches 84.4%.
- On ADE20K with Semantic FPN, GLNet-4G/16G achieve 49.6/51.3 mIoU, improving previous best SG-Former-S/M by 0.6/1.2 mIoU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use GLNet-4G instead of the strongest listed comparable supervised baseline | ImageNet-1K classification · ImageNet-1K standard supervised | top-1 accuracy: 83.5 -> 83.7 | Helped |
| use GLNet-9G instead of MaxViT-S | ImageNet-1K classification · ImageNet-1K standard supervised | top-1 accuracy: 84.4 -> 84.5 | Helped |
| use GLNet-16G instead of MaxViT-B | ImageNet-1K classification · ImageNet-1K standard supervised | top-1 accuracy: 84.9 -> 85.0 | Helped |
| add distillation supervision to GLNet-4G | GLNet-4G · ImageNet-1K classification | top-1 accuracy: 83.7% to 84.4% | Helped |
| use GLNet-4G instead of BiFormer-S | COCO 2017 RetinaNet 1× · COCO 2017 RetinaNet 1× | box mAP: 45.9 -> 47.1 | Helped |
| use GLNet-9G instead of BiFormer-B | COCO 2017 RetinaNet 1× · COCO 2017 RetinaNet 1× | box mAP: 47.1 -> 47.7 | Helped |
| use GLNet-9G instead of SG-Former-M | COCO 2017 Mask R-CNN 3× + MS · COCO 2017 Mask R-CNN 3× + MS | box mAP: 50.5 -> 51.0 | Helped |
| use GLNet-9G instead of SG-Former-M | ADE20K Semantic FPN · ADE20K Semantic FPN 80k | mIoU: 50.1 -> 51.3 | Helped |
| add the local branch to the global branch (ablation) | GLNet-STL · ImageNet-1K classification | top-1 accuracy: 78.0 -> 82.5 | Helped |
| add the global branch to the local branch (ablation) | GLNet-STL · ImageNet-1K classification | top-1 accuracy: 81.8 -> 82.5 | Helped |
| use parallel rather than sequential local-global branches (ablation) | GLNet-STL · ImageNet-1K classification | top-1 accuracy: 80.6 -> 82.5; 79.6 -> 82.5 | Helped |
| use depth-wise convolution rather than window MHSA in the local branch (ablation) | GLNet-STL · ImageNet-1K classification | throughput: 660.9 im/s -> 835.9 im/s | Helped |
Surprising findings
- Using only a global branch with 64 slots and no local convolution drops accuracy to 78.0%, while a local-only branch reaches 81.8%, and replacing the local depth-wise conv with window MHSA lowers throughput.
- Soft clustering trained only with ImageNet classification labels yields semantic grouping of foreground objects without dense supervision.
Limitations
- Uses a static number of semantic slots (64) for all images, which can produce redundant slots.
- Still relies on many hardware-inefficient depth-wise convolutions with low arithmetic intensity.
- Throughput gains are smaller than FLOPs gains and the advanced designs reduce throughput substantially (e.g., 835.9 to 541.2 im/s).
Experimental setup
Datasets: ImageNet-1k COCO 2017 ADE20K
Benchmarks: ImageNet-1k COCO 2017 ADE20K
Metrics: top-1 accuracy box mAP mask mAP mIoU throughput (im/s) FLOPs
Baselines: Swin-T DAT-T Swin-ACMix-T Flatten-Swin-T NAT-T MaxViT-T CSWin-T SG-Former-S SMT-S BiFormer-S
Builds on: Swin Transformer ACMix SoftMoE Feature Pyramid Network timm
Closest papers by meaning
- Bridging the Divide: Reconsidering Softmax and Linear Attention (NeurIPS 2024 · similarity 0.88)
- Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.87)
- Large Vision-Language Models Get Lost in Attention (ICML 2026 · similarity 0.86)
- MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models (ICLR 2025 · similarity 0.86)
- MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughly (NeurIPS 2025 · similarity 0.86)
- LSNet: See Large, Focus Small (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.