Research map / Detection and segmentation / CNN features
SSA-Seg: Semantic and Spatial Adaptive Pixel-level Classifier for Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SSA-Seg, a pixel-level classifier that adapts fixed class prototypes toward semantic and spatial centers of each test image (guided by coarse masks) and trains with online multi-domain distillation from a ground-truth-guided teacher branch.
Problem
Fixed-prototype pixel-level softmax classifiers for semantic segmentation suffer feature deviation in the semantic domain (large intra-class variance across images) and information loss in the spatial domain, leading to blurred boundaries and poor small-target recognition.
Approach
Adapt fixed class prototypes per test image using coarse-mask-guided semantic centers and spatial centers (with conditional position encoding), classify using both adapted domains, and constrain adaptation with an online teacher branch that uses ground-truth masks and multi-domain distillation (response, semantic, spatial).
Key results
- SeaFormer-L + SSA-Seg reaches 45.36% mIoU on ADE20K (+3.00 over 42.36) with only 0.1G extra FLOPs and 0.5 ms extra latency.
- UperNet-Swin-T + SSA-Seg improves ADE20K mIoU from 44.14 to 47.56 (+3.42).
- SegNeXt-T + SSA-Seg reaches 38.91% mIoU on COCO-Stuff-10K and 52.58% on PASCAL-Context.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SSA-Seg to SeaFormer-L | SeaFormer-L · ADE20K | mIoU: 42.36 -> 45.36 | Helped |
| Add SSA-Seg to SegNeXt-T | SegNeXt-T · COCO-Stuff-10K | mIoU: 36.39 -> 38.91 | Helped |
| Add SSA-Seg to SegNeXt-T | SegNeXt-T · PASCAL-Context | mIoU: 50.35 -> 52.58 | Helped |
| Remove entropy weighting from response domain distillation (ablation) | SSA-Seg · ADE20K | mIoU: 45.36 -> 44.82 | Hurt |
| Remove boundary mask from response domain distillation (ablation) | SSA-Seg · ADE20K | mIoU: 45.36 -> 44.77 | Hurt |
| Remove category mask from response domain distillation (ablation) | SSA-Seg · ADE20K | mIoU: 45.36 -> 45.11 | Hurt |
| Apply softmax over spatial dimensions rather than classes when generating the spatial center (ablation) | SSA-Seg · ADE20K | mIoU: 44.86 -> 45.36 | Helped |
| Use CPVT rather than sinusoidal position encoding (ablation) | SSA-Seg · ADE20K | mIoU: 44.18 -> 45.36 | Helped |
| Use CPVT rather than learnable absolute position encoding (ablation) | SSA-Seg · ADE20K | mIoU: 44.02 -> 45.36 | Helped |
| Add semantic prototype adaptation (SEPA) (ablation) | SeaFormer-L baseline · ADE20K | mIoU: 42.36 -> 42.88 | Helped |
| Add spatial prototype adaptation (SPPA) (ablation) | SeaFormer-L baseline · ADE20K | mIoU: 42.36 -> 42.76 | Helped |
| Add SEPA with ground-truth and semantic-domain distillation (ablation) | SeaFormer-L baseline · ADE20K | mIoU: 42.36 -> 44.26 | Helped |
Surprising findings
- Adding semantic and spatial prototype adaptation without the corresponding distillation losses yields only marginal gains (+0.91 mIoU), and unconstrained prototype offsets can move away from the image features.
Limitations
- Not compatible with mask-level classification methods (e.g., Mask2Former, SegViT), limiting applicability.
- Gains on general large models are smaller (e.g., +0.28 to +1.01 mIoU for ViT-Adapter-L), and it relies on more advanced pixel-level baselines for general segmentation.
- Training requires about 1.1x training time due to the teacher branch.
Experimental setup
Datasets: ADE20K PASCAL-Context COCO-Stuff-10K
Benchmarks: ADE20K PASCAL-Context COCO-Stuff-10K
Metrics: mIoU FLOPs latency parameters
Baselines: SeaFormer-L SeaFormer-B SegNeXt-T AFFormer-B UperNet SegFormer OCRNet ViT-Adapter-L CAC GMMSeg ProtoSeg DNC
Builds on: CPVT conditional position encoding Context-aware Classifier (CAC) GMMSeg SeaFormer SegNeXt
Closest papers by meaning
- Exploring Simple Open-Vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.88)
- PSDPM: Prototype-based Secondary Discriminative Pixels Mining for Weakly Supervised Semantic Segmentation (CVPR 2024 · similarity 0.86)
- SemiDAViL: Semi-supervised Domain Adaptation with Vision-Language Guidance for Semantic Segmentation (CVPR 2025 · similarity 0.86)
- Improving Semi-Supervised Semantic Segmentation with Sliced-Wasserstein Feature Alignment and Uniformity (CVPR 2025 · similarity 0.86)
- SED: A Simple Encoder-Decoder for Open-Vocabulary Semantic Segmentation (CVPR 2024 · similarity 0.86)
- Semantic Library Adaptation: LoRA Retrieval and Fusion for Open-Vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.