Research map / Detection and segmentation / CNN features
Multi-Scale Representations by Varying Window Attention for Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes varying window attention (VWA), which enlarges the context window of local window attention at no extra cost via pre-scaling (DOPE), and builds VWFormer, a multi-scale decoder for semantic segmentation.
Problem
Multi-scale representations learned by existing semantic segmentation paradigms (atrous convs, adaptive pooling, hierarchical backbones with MSDs) suffer from scale inadequacy and field inactivation, as shown by effective receptive field analysis.
Approach
Disentangle local window attention into a query window and a larger context window (ratio R) so the query sees multiple receptive fields; a pre-scaling step (DOPE, channel reduction before enlarging context) plus copy-shift padding keeps the cost equal to LWA plus one linear mapping.
Key results
- VWFormer with Swin-B and UperNet-style comparison: ADE20K 52.5 / 53.5 mIoU at 120 GFLOPs vs. UperNet Swin-B 50.8 / 52.4 at 306 GFLOPs.
- Mask2Former with VWFormer improves Swin-L from 56.1 to 56.5 mIoU (single-scale) at similar cost, and combined with deformable attention reaches 56.9 mIoU for Swin-L.
- Using zero padding instead of copy-shift padding drops Swin-B ADE20K mIoU from 52.5 to 52.0 (single-scale).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the MLP decoder with VWFormer | SegFormer with MiT-B5 · ADE20K | mIoU: 51.0 -> 52.0 | Helped |
| Replace UperNet with VWFormer | Swin-B segmentation · ADE20K | mIoU: 50.8 -> 52.5 | Helped |
| Replace FPN with VWFormer | MaskFormer with Swin-B · ADE20K | mIoU: 52.7 -> 53.8 | Helped |
| Replace Deformable Attention with VWFormer | Mask2Former with Swin-B · ADE20K | mIoU: 53.9 -> 54.6 | Helped |
| Add VWFormer alongside Deformable Attention | Mask2Former with Swin-B · ADE20K | mIoU: 54.6 -> 55.2 | Helped |
| Add each of the R=2, R=4, and R=8 VWA scales (ablation) | VWFormer · ADE20K | mIoU: 51.8 -> 52.5; 51.7 -> 52.5; 51.9 -> 52.5 | Helped |
| Add an R=1 LWA branch while replacing R=2 VWA (ablation) | VWFormer · ADE20K | mIoU: 52.5 -> 52.1; 52.5 -> 51.9 | Hurt |
| Use post-scaling PE instead of pre-scaling DOPE → PE (ablation) | VWFormer · ADE20K | mIoU: 52.5 -> 52.3 | No clear effect |
| Use post-scaling average pooling (ablation) | VWFormer · ADE20K | mIoU: 52.5 -> 51.7 | Hurt |
| Remove context-window rescaling (ablation) | VWFormer · ADE20K | mIoU: 52.5 -> 52.4 | No clear effect |
| Use CSP instead of zero padding (ablation) | VWFormer · ADE20K | mIoU: 52.0 -> 52.5 | Helped |
| Add low-level enhancement (ablation) | VWFormer · ADE20K | mIoU: 51.8 -> 52.5 | Helped |
Surprising findings
- Adding an R=1 (plain LWA) branch does not help because the 1x1 short path already provides a very local receptive field.
- Post-scaling with patch embedding costs more FLOPs (210G) than pre-scaling (120G) and yields no better mIoU.
Limitations
- Complexity analysis and efficiency gains are tied to specific backbones and decoder settings (Swin, ConvNeXt, MiT).
- Many baseline numbers are copied from original papers rather than re-run under identical settings.
Experimental setup
Datasets: ADE20K Cityscapes COCOStuff-164K
Benchmarks: ADE20K Cityscapes COCOStuff-164K
Metrics: mIoU mIoU multi-scale FLOPs params memory footprint FPS
Baselines: SegFormer MLP-decoder UperNet FPN MaskFormer Mask2Former Deformable attention PSPNet DeepLabV3 DeepLabV3+ DenseASPP APCNet DMNet
Builds on: local window attention Swin Transformer SegFormer UperNet Mask2Former
Closest papers by meaning
- Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models (CVPR 2024 · similarity 0.87)
- ScaleMatch: Multi-scale Consistency Enhancement for Semi-supervised Semantic Segmentation (AAAI 2025 · similarity 0.87)
- Integrating Low-Level Visual Cues for Enhanced Unsupervised Semantic Segmentation (AAAI 2025 · similarity 0.87)
- CLIP as RNN: Segment Countless Visual Concepts without Training Endeavor (CVPR 2024 · similarity 0.86)
- Test-Time Adaptation of Vision-Language Models for Open-Vocabulary Semantic Segmentation (NeurIPS 2025 · similarity 0.86)
- Pixels to Tokens: Token Space Efficient Active Learning for Low-Budget Semantic Segmentation (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.