Research map / Detection and segmentation / CNN features
Native Segmentation Vision Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Native Segmentation Vision Transformer (NaSeViT), a backbone whose content-aware spatial grouping layers replace uniform downsampling so segmentation masks emerge natively, with strong zero-shot and mask-supervised results.
Problem
Uniform downsampling in hierarchical vision backbones ignores image content and misaligns features with boundaries, forcing segmentation decoders to compensate; prior grouping-based backbones are quadratic in cost or use non-differentiable clustering.
Approach
Replace strided/pooling downsampling with a differentiable, iterative (K-means/slot-attention-like) spatial grouping layer that is local in early stages and dense in the final stage, so stacked assignment matrices yield hierarchical pixel-to-token masks without segmentation heads.
Key results
- Native NaSeViT-T reaches 49.7 mIoU on ADE20k with a UperNet-free native head, +2.6 over NAT-T with UperNet (47.1) at about 12% of its FLOPs.
- Zero-shot text-supervised NaSeViT-B (CC3M+CC12M) averages 31.9 mIoU across six datasets without postprocessing, versus 31.8 for CoDe which uses PAMR; adding RedCaps12M raises it to 33.1.
- NaSeViT-B with Mask2Former reaches 57.1 PQ on COCO-panoptic (Swin-B 56.4), and NaSeViT-L reaches 58.1 PQ.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train with an additional 12M RedCaps12M image-text pairs | NaViT-B · Zero-shot semantic segmentation (six-dataset Avg.) | mIoU: 31.9 -> 33.1 | Helped |
| use native grouping-based masks instead of uniform-downsampling NAT with UperNet | Tiny backbone on ADE20k · ADE20k val semantic segmentation | mIoU: 47.1 -> 49.7 | Helped |
| replace Swin-T with NaViT-T as the Mask2Former backbone | Mask2Former on ADE20k · ADE20k val semantic segmentation | mIoU: 47.2 -> 51.3 | Helped |
| replace Swin-T with NaViT-T for native masks | Native segmentation on COCO · COCO val panoptic segmentation | PQ: 47.7 -> 49.2 | Helped |
| replace uniform downsampling with grouping at S1, S2, and S3 (ablation) | NaViT · ADE20k native semantic segmentation | mIoU: 41.3 -> 49.7 | Helped |
| replace uniform downsampling with grouping at S1, S2, and S3 (ablation) | NaViT · Pascal VOC zero-shot segmentation | mIoU: 40.1 -> 57.3 | Helped |
| use dense rather than local grouping in the final stage (ablation) | NaViT · ADE20k native semantic segmentation | mIoU: 47.3 -> 49.7 | Helped |
| use skip connections instead of a GRU to update groups (ablation) | NaViT grouping layer · ADE20k native semantic segmentation | mIoU: 44.9 -> 49.7 (+4.8) | Helped |
| use relative instead of absolute positional encoding (ablation) | NaViT grouping layer · ADE20k native semantic segmentation | mIoU: 48.8 (-0.9) -> 49.7 | Helped |
| initialize groups with strided convolution rather than learned nn.Embedding vectors (ablation) | NaViT grouping layer · ADE20k native semantic segmentation | mIoU: 47.2 (-2.5) -> 49.7 | Helped |
| use the proposed grouping-layer design rather than the Slot Attention variant (ablation) | NaViT grouping layer · ADE20k native semantic segmentation | mIoU: 43.6 (-6.1) -> 49.7 | Helped |
| use the proposed grouping-layer design rather than the Slot Attention variant (ablation) | NaViT grouping layer · Pascal VOC zero-shot segmentation | mIoU: 52.3 (-5.0) -> 57.3 | Helped |
Surprising findings
- Native masks without any segmentation head outperform NAT with UperNet on ADE20k, despite lower compute.
- Local grouping in the last stage (instead of dense) significantly decreases performance in both supervised and zero-shot settings.
Limitations
- Grouping layers add computational overhead over strided convolutions, which is amortized only in the native setup.
- Gains are larger for semantic than panoptic segmentation, possibly due to ImageNet pre-training biases against instance separation.
- Zero-shot results still lag CLIP-based methods on ADE20k and COCO-Stuff with many classes.
Experimental setup
Datasets: ImageNet-1k ImageNet-22k CC3M CC12M RedCaps12M ADE20k COCO-panoptic Pascal VOC Pascal Context COCO-Stuff COCO-Object Cityscapes
Benchmarks: ADE20k COCO-panoptic Pascal VOC Pascal Context COCO-Stuff COCO-Object ADE20k zero-shot Cityscapes ImageNet-1k
Metrics: mIoU PQ top-1 accuracy FPS FLOPs parameters
Baselines: Swin Transformer NAT GroupViT ClusterFormer TCFormer CoDe TCL SimSeg UperNet MaskFormer Mask2Former CAST
Builds on: Slot Attention NAT Mask2Former GroupViT SLIC
Closest papers by meaning
- GroupViT: Semantic Segmentation Emerges From Text Supervision (CVPR 2022 · similarity 0.87)
- Rethinking Semantic Segmentation From a Sequence-to-Sequence Perspective With Transformers (CVPR 2021 · similarity 0.87)
- Hierarchy-Agnostic Unsupervised Segmentation: Parsing Semantic Image Structure (NeurIPS 2024 · similarity 0.87)
- Learning Hierarchical Image Segmentation For Recognition and By Recognition (ICLR 2024 · similarity 0.86)
- Segment Anything without Supervision (NeurIPS 2024 · similarity 0.86)
- SED: A Simple Encoder-Decoder for Open-Vocabulary Semantic Segmentation (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.