Research map / Detection and segmentation / Open-set and few-shot detection
Open-World Semantic Segmentation Including Class Similarity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ContMAV, a single-encoder, double-decoder CNN that jointly performs closed-world semantic segmentation, anomaly segmentation, and open-world novel class discovery with class similarity, without extra training data.
Problem
Open-world semantic segmentation: segmenting pixels of unseen object categories in camera images while keeping closed-world accuracy, distinguishing between different novel classes, and estimating their similarity to known classes.
Approach
A semantic decoder pushes features of each known class toward a learned class mean with a feature loss and models per-class Gaussians, while a contrastive decoder combines contrastive and objectosphere losses; the two outputs are fused into unknown scores, novel classes are grown by distance thresholding in feature space, and similarity is the Gaussian fit score argmax.
Key results
- On SegmentMeIfYouCan, ContMAV ranked first in FPR95 (3.8%), PPV (61.9%), and mean F1 (63.6%) on the leaderboard at submission time.
- On BDDAnomaly, ContMAV reached 96.1% AUPR and 6.9% FPR95 versus 5.4% AUPR and 14.0% FPR95 for the best baseline MaxLogit.
- Open-world segmentation on BDDAnomaly reduced known-class mIoU only slightly (Cityscapes 71.1 to 70.8), and class-similarity accuracy for motorcycle reached 58.9% versus 12.5% for the baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the feature loss to the semantic decoder (ablation) | ContMAV · BDDAnomaly, novel class discovery | mIoU [%]: 48.1 -> 62.4; 53.8 -> 62.2; 39.9 -> 56.8 | Helped |
| add the contrastive decoder (ablation) | ContMAV anomaly segmentation · BDDAnomaly | AUPR [%]: 91.8 -> 94.1 | Helped |
| add the contrastive decoder (ablation) | ContMAV anomaly segmentation · BDDAnomaly | FPR95 [%]: 69.7 -> 6.9 | Helped |
| use Gaussian post-processing instead of minimum-distance post-processing (ablation) | ContMAV anomaly segmentation · BDDAnomaly | AUPR [%]: 94.8 -> 96.1 | Helped |
| use Gaussian post-processing for class similarity instead of maximum activation (ablation) | ContMAV · BDDAnomaly* | Accuracy [%]: 39.9 -> 57.8; 27.6 -> 48.6 | Helped |
| add the contrastive decoder to Gaussian class-similarity post-processing (ablation) | ContMAV · BDDAnomaly* | Accuracy [%]: 57.8 -> 58.9; 48.6 -> 49.9 | No clear effect |
| use ContMAV open-world segmentation instead of the closed-world model | semantic segmentation · Cityscapes | mIoU: 71.1 -> 70.8 | Hurt |
| use ContMAV open-world segmentation instead of the closed-world model | semantic segmentation · BDDAnomaly | mIoU: 64.1 -> 62.8 | Hurt |
| use ContMAV instead of MaxLogit | anomaly segmentation · BDDAnomaly | AUPR [%]: 5.4 -> 96.1 | Helped |
| use ContMAV instead of MaxLogit | anomaly segmentation · BDDAnomaly | FPR95 [%]: 14.0 -> 6.9 | Helped |
| use Gaussian class similarity instead of the learned lookup-table classifier | ContMAV · BDDAnomaly* | Accuracy [%]: 12.5 -> 58.9; 9.8 -> 49.9 | Helped |
Surprising findings
- Thresholding softmax alone performs poorly for anomaly segmentation, while the contrastive decoder alone outperforms softmax thresholding.
- A learned linear-layer classifier on pre-softmax features does not generalize well to unknown classes for similarity, performing worse than simply taking the maximum-activation class.
Limitations
- Unknown-class similarity relies on a manually created lookup table for evaluation.
- Anomaly segmentation is harder on BDDAnomaly due to small objects such as bicycles and motorcycles.
- Performance on distinguishing novel classes (mIoU 62.4 / 62.2 / 56.8) remains below closed-world upper bound.
Experimental setup
Datasets: Cityscapes BDDAnomaly BDDAnomaly*
Benchmarks: SegmentMeIfYouCan BDDAnomaly BDDAnomaly*
Metrics: mIoU AUPR FPR95 sIoU PPV mean F1 accuracy
Baselines: DenseHybrid RbA Maskomaly MaxSoftmax Background MC Dropout Confidence MaxLogit
Builds on: objectosphere loss SwiftNet ERFNet NonBottleneck-1D PSPNet pyramid pooling contrastive loss
Closest papers by meaning
- Training-free Open-Vocabulary Semantic Segmentation via Diverse Prototype Construction and Sub-region Matching (AAAI 2025 · similarity 0.87)
- PDF: A Probability-Driven Framework for Open World 3D Point Cloud Semantic Segmentation (CVPR 2024 · similarity 0.87)
- A Dataset for Semantic Segmentation in the Presence of Unknowns (CVPR 2025 · similarity 0.86)
- OpenWorldSAM: Extending SAM2 for Universal Image Segmentation with Language Prompts (NeurIPS 2025 · similarity 0.86)
- Distilling Spectral Graph for Object-Context Aware Open-Vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.85)
- v-CLR: View-Consistent Learning for Open-World Instance Segmentation (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.