Research map / Detection and segmentation / Open-set and few-shot detection
Generalize or Detect? Towards Robust Semantic Segmentation Under Multiple Distribution Shifts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes coherent generative augmentation (ControlNet-based, with domain and semantic shifts generated jointly) plus a two-stage, noise-aware training with a learnable semantic-exclusive uncertainty function, to jointly improve anomaly segmentation and domain generalization.
Problem
Semantic segmentation models must both detect novel-object anomalies (semantic shifts) and generalize to novel domains (covariate shifts), but existing OOD-detection and domain-generalization methods each fail on the other type of shift, often confusing domain shifts with anomalies.
Approach
Generate training images that jointly contain pasted anomaly objects and varied domain/weather/time changes via a semantic-to-image ControlNet, then learn a semantic-exclusive uncertainty function with a relative contrastive loss and fine-tune the feature extractor with pixel-wise small-loss sample selection.
Key results
- On RoadAnomaly with Mask2Former, achieves 90.17 AP and 7.54 FPR95, versus 85.42 AP for RbA.
- On ACDC-POC with Mask2Former, reaches 90.42 AP and 0.46 FPR95, versus 73.77 AP for Mask2Anomaly.
- On MUAD with Mask2Former, reaches 45.65 AP versus 39.32 for Mask2Anomaly, with mIoU 28.44 vs 23.43.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace Mask2Anomaly's default COCO OOD data with CG-Aug (ablation) | Mask2Anomaly · RoadAnomaly | AP: 79.70 -> 85.47 | Helped |
| Replace Mask2Anomaly's training strategy with the proposed training strategy (ablation) | Mask2Anomaly + CG-Aug · RoadAnomaly | AP: 85.47 -> 90.17 | Helped |
| Add domain-shift data to semantic-shift-only augmentation (ablation) | CG-Aug variants · RoadAnomaly | AP: 83.66 -> 87.64 | Helped |
| Generate domain and semantic shifts jointly rather than separately (ablation) | CG-Aug variants · RoadAnomaly | AP: 89.08 -> 90.17 | Helped |
| Add a learnable uncertainty function instead of using a fixed energy function (ablation) | Training pipeline · SMIYC-RA Val | AP: 89.34 -> 93.82 | Helped |
| Use relative contrastive loss instead of absolute contrastive loss (ablation) | Training pipeline · SMIYC-RA Val | AP: 91.01 -> 93.82 | Helped |
| Add noise-aware sample selection (ablation) | Training pipeline · SMIYC-RA Val | AP: 91.64 -> 93.82 | Helped |
| Add noise-aware sample selection (ablation) | Training pipeline · MUAD | mIoU: 31.73 -> 31.33 | Hurt |
| Use CG-Aug with RPL's original training strategy (ablation) | RPL · SMIYC-RA Val | AP: 88.55 -> 83.50 | Hurt |
| Use the proposed training strategy with CG-Aug instead of RPL's training strategy (ablation) | RPL + CG-Aug · SMIYC-RA Val | AP: 83.50 -> 93.82 | Helped |
| Increase loss-margin scale from x 1 to x 10 (ablation) | Proposed method · SMIYC-RA Val | AP: 93.82 -> 95.73 | Helped |
Surprising findings
- Replacing RPL's OOD data with CG-Aug gives only a marginal improvement (71.61 to 72.46 AP on RoadAnomaly), suggesting its loss and training design do not exploit the generated data well.
- Domain generalization methods like RobustNet and RuleAug can degrade OOD detection below the baseline, and OOD methods like PEBAL can hurt known-class segmentation.
Limitations
- Performance depends on the quality of the generative model; generation failures occur for remote scenes, small objects, and text.
- Performance saturates as the amount of generated data increases.
- Some classes (e.g., vegetation) drop slightly in mIoU, likely due to poor generation quality.
Experimental setup
Datasets: Cityscapes COCO ADE20K AnomalyMix ACDC
Benchmarks: RoadAnomaly SMIYC-RA21 SMIYC-RO21 ACDC-POC MUAD ACDC
Metrics: AUROC AP FPR95 mIoU mAcc
Baselines: Maximum softmax ODIN Mahalanobis SynBoost Maximized entropy PEBAL Dense Hybrid RPL+CoroCL Mask2Anomaly RbA M2F-EAM RuleAug
Builds on: PEBAL Mask2Anomaly RPL ControlNet Mask2Former
Closest papers by meaning
- Exploring Semantic Consistency and Style Diversity for Domain Generalized Semantic Segmentation (AAAI 2025 · similarity 0.88)
- A Simple Recipe for Language-guided Domain Generalized Segmentation (CVPR 2024 · similarity 0.88)
- Leveraging Depth and Language for Open-Vocabulary Domain-Generalized Semantic Segmentation (NeurIPS 2025 · similarity 0.87)
- ImageNet-OOD: Deciphering Modern Out-of-Distribution Detection Algorithms (ICLR 2024 · similarity 0.87)
- Collaborating Foundation Models for Domain Generalized Semantic Segmentation (CVPR 2024 · similarity 0.87)
- Open-Set Domain Adaptation for Semantic Segmentation (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.