Research map / Detection and segmentation / CNN features
Rethinking Decoders for Transformer-based Semantic Segmentation: A Compression Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives DEPICT, a white-box fully attentional Transformer decoder for semantic segmentation from a PCA/coding-rate compression perspective, outperforming Segmenter on ADE20K with far fewer parameters.
Problem
Transformer decoders for semantic segmentation are empirically designed and lack theoretical justification or interpretation, which hinders principled improvements.
Approach
Interprets Transformer decoders as PCA-style compression: self-attention (MSSA) unrolls coding-rate maximization to build an ideal principal subspace, and cross-attention (MSCA) finds a low-rank approximation giving orthonormal class embeddings; dot-product yields masks. The resulting decoder drops the FFN and uses far fewer parameters.
Key results
- DEPICT-SA with ViT-L at 640x640 reaches 52.9/54.3 mIoU (ss/ms) on ADE20K versus Segmenter's 51.8/53.6, using about 2M vs 28M parameters.
- On Cityscapes and Pascal Context (ViT-L), DEPICT-SA is slightly below Segmenter (78.8 vs 79.1 ss; 57.9 vs 58.1 ss).
- Under parameter perturbations, DEPICT-SA stays robust while Segmenter collapses (e.g., orthogonal transform of P drops Segmenter performance by over 40 mIoU points).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DEPICT-SA instead of Segmenter | ADE20K validation, ViT-L · ADE20K validation | mIoU (ss): 51.8 -> 52.9 | Helped |
| Use DEPICT-SA instead of Segmenter | ADE20K validation, ViT-L · ADE20K validation | mIoU (ms): 53.6 -> 54.3 | Helped |
| Use DEPICT-CA instead of Segmenter | ADE20K validation, ViT-L · ADE20K validation | mIoU (ss): 51.8 -> 52.8 | Helped |
| Use DEPICT-SA instead of Segmenter | Cityscapes validation, ViT-L · Cityscapes validation | mIoU (ss): 79.1 -> 78.8 | Hurt |
| Use DEPICT-SA instead of Segmenter | Cityscapes validation, ViT-L · Cityscapes validation | mIoU (ms): 81.3 -> 81.0 | Hurt |
| Use DEPICT-SA instead of Segmenter | Pascal Context validation, ViT-L · Pascal Context validation | mIoU (ss): 58.1 -> 57.9 | Hurt |
| Use DEPICT-SA instead of Segmenter | Pascal Context validation, ViT-L · Pascal Context validation | mIoU (ms): 59.0 -> 58.6 | Hurt |
| Use 3 heads rather than 5 (ablation) | MSSA + ISTA, ViT-S · ADE20K | mIoU: 45.6 -> 45.6 | No clear effect |
| Use 3 layers rather than 1 (ablation) | MSSA + ISTA, ViT-S · ADE20K | mIoU: 45.2 -> 45.8 | Helped |
| Set dropout to 0.1 rather than 0.0 (ablation) | MSSA + ISTA, ViT-S · ADE20K | mIoU: 45.6 -> 45.2 | Hurt |
| Add ISTA to MSSA (ablation) | MSSA only, ViT-S · ADE20K | mIoU: 45.4 -> 45.6 | Helped |
| Add MSSA to ISTA (ablation) | ISTA only, ViT-S · ADE20K | mIoU: 44.9 -> 45.6 | Helped |
Surprising findings
- Removing the FFN block from the Transformer decoder does not hurt and the decoder remains competitive, while naively using MSSA without the derived design degrades performance.
- DEPICT slightly underperforms Segmenter on Cityscapes and Pascal Context, attributed to small training set sizes.
Limitations
- Gains are modest over Segmenter (about 1 mIoU) and DEPICT trails MaskFormer, which uses a pixel decoder, hierarchical backbone, and mask classification.
- Slightly worse than Segmenter on Cityscapes and Pascal Context, possibly due to small dataset size.
- Uses a relatively simple implementation with non-hierarchical features and does not explore mask classification.
Experimental setup
Datasets: ADE20K Cityscapes Pascal Context
Benchmarks: ADE20K Cityscapes Pascal Context
Metrics: mIoU parameters FLOPs
Baselines: Segmenter MaskFormer
Builds on: CRATE MCR^2 Segmenter DETR ReduNet
Closest papers by meaning
- FeedFormer: Revisiting Transformer Decoder for Efficient Semantic Segmentation (AAAI 2023 · similarity 0.91)
- SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers (NeurIPS 2021 · similarity 0.91)
- Rethinking Semantic Segmentation From a Sequence-to-Sequence Perspective With Transformers (CVPR 2021 · similarity 0.89)
- Head-Free Lightweight Semantic Segmentation with Linear Transformer (AAAI 2023 · similarity 0.89)
- Generalized Decoding for Pixel, Image, and Language (CVPR 2023 · similarity 0.89)
- Dense Decoder Shortcut Connections for Single-Pass Semantic Segmentation (CVPR 2018 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.