Research map / Detection and segmentation / Open-set and few-shot detection
CSL: Class-Agnostic Structure-Constrained Learning for Segmentation Including the Unseen
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CSL, a class-agnostic structure-constrained plug-in framework (distillation or inference-only fusion) with soft assignment and mask split to improve OOD, zero-shot and domain-adaptive semantic segmentation.
Problem
Segmenting unseen classes for out-of-distribution (OOD) segmentation, zero-shot semantic segmentation (ZS3), and domain adaptation without OOD training data; existing class-agnostic Mask2Former variants give noisy, incomplete region proposals under hard assignment.
Approach
Class-agnostic region prototypes with a soft (pixel-wise) assignment that combines region scores and validity scores, a mask-split preprocessing that breaks semantic masks into disconnected components to reduce seen-class bias, and a structure-constrained fusion that combines region-averaged and pixel-wise scores; it can be used either via distillation from a base teacher (scheme 1) or applied at
Key results
- On SMIYC-AT test, ObsNet v2 + CSL2 reduces FPR95 from 26.69 to 7.16 and raises AP from 75.44 to 80.1 without OOD training data.
- On PASCAL VOC 2012 ZS3, ZegFormer + CSL2 improves unseen mIoU from 63.6 to 68.6.
- On Synscapes2Cityscapes, CSL2 raises DAFormer mIoU from 65 to 70 (+4 points), with source-only improving from 43 to 47.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CSL₂ to ObsNet v2 | ObsNet v2 · SMIYC (AT)-test | AP: 75.44 -> 80.1 | Helped |
| Add CSL₂ to ObsNet v2 | ObsNet v2 · SMIYC (AT)-test | FPR₉₅: 26.69 -> 7.16 | Helped |
| Add CSL₂ to ZegFormer | ZegFormer · COCO-stuff | hIoU: 34.8 -> 36.9 | Helped |
| Add CSL₂ to ZSSeg | ZSSeg · COCO-stuff | hIoU: 37.8 -> 39.2 | Helped |
| Add CSL₂ to ZegCLIP | ZegCLIP · COCO-stuff | hIoU: 40.8 -> 41.6 | Helped |
| Add CSL₂ to Source Only | Source Only · Synscapes2Cityscapes | mIoU: 43 -> 47 | Helped |
| Add CSL₂ to AdvEnt | AdvEnt · Synscapes2Cityscapes | mIoU: 52 -> 56 | Helped |
| Add CSL₂ to DAFormer | DAFormer · Synscapes2Cityscapes | mIoU: 65 -> 70 | Helped |
| Add class-agnostic training (ablation) | CSL region proposal generator · COCO-stuff | mIoU: 45.99 -> 49.17 | Helped |
| Replace hard assignment with soft assignment (ablation) | CA-trained region proposal generator · COCO-stuff | mIoU: 49.17 -> 51.34 | Helped |
| Add class-agnostic training (ablation) | CSL region proposal generator · COCO-stuff OOD | OOD-mIoU: 43.83 -> 46.56 | Helped |
| Replace hard assignment with soft assignment (ablation) | CA-trained region proposal generator · COCO-stuff OOD | OOD-mIoU: 46.56 -> 48.85 | Helped |
Surprising findings
- Replacing the SAM-based region proposals with CSL proposals gave only a small difference (about 1.7% AUPR on average), suggesting the constraining factor is per-region classification accuracy rather than segmentation quality.
Limitations
- Scheme 2 inference is roughly 33% slower than scheme 1 on average.
- Performance is limited by per-region classification accuracy according to the authors' analysis.
Experimental setup
Datasets: Cityscapes COCO-stuff PASCAL VOC 2012 Synscapes SMIYC Road Anomaly LostAndFound
Benchmarks: SMIYC Anomaly Track SMIYC Obstacle Track LostAndFound NoKnown Road Anomaly COCO-stuff PASCAL VOC 2012 Synscapes2Cityscapes
Metrics: AUPR FPR95 sIoU gt mean F1 AUROC AP mIoU hIoU
Baselines: SML Image Resynthesis ObsNet ObsNet v2 DaCUP NFlowJS ZegFormer ZSSeg ZegCLIP AdvEnt DAFormer PEBAL
Builds on: Mask2Former CLIP ObsNet ZegFormer
Closest papers by meaning
- Open-Set Domain Adaptation for Semantic Segmentation (CVPR 2024 · similarity 0.87)
- SemiDAViL: Semi-supervised Domain Adaptation with Vision-Language Guidance for Semantic Segmentation (CVPR 2025 · similarity 0.87)
- Distilling Spectral Graph for Object-Context Aware Open-Vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.86)
- S3A: Towards Realistic Zero-Shot Classification via Self Structural Semantic Alignment (AAAI 2024 · similarity 0.86)
- Test-Time Adaptation of Vision-Language Models for Open-Vocabulary Semantic Segmentation (NeurIPS 2025 · similarity 0.86)
- Semantic Library Adaptation: LoRA Retrieval and Fusion for Open-Vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.