Research map / Detection and segmentation / Open-set and few-shot detection
Collaborating Foundation Models for Domain Generalized Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CLOUDS, a collaborative foundation-model system for domain generalized semantic segmentation that combines a frozen CLIP backbone with Mask2Former decoder, Stable Diffusion-generated content (prompts from Llama-2), and SAM-refined pseudo labels for self-training.
Problem
Domain generalized semantic segmentation (DGSS): training on a labeled source domain (GTA or SYNTHIA) and generalizing to unseen real and adverse-weather domains, where existing domain randomization only diversifies style, not content.
Approach
Integrates multiple foundation models: CLIP as a frozen feature extractor, LLM-diversified prompts driving a text-to-image diffusion model to generate content-diverse images, and SAM point-prompted to refine teacher pseudo labels for an EMA student-teacher self-training loop.
Key results
- On GTA to Cityscapes/BDD/Mapillary with ConvNext-L, CLOUDS reaches 61.5% average mIoU, versus 55.9% for HRDA with MiT-B5 and 57.0% for FC-CLIP (COCO-trained).
- With ResNet-50 on GTA, CLOUDS achieves 53.3% average mIoU, +8.1% over MoDify (45.2% reported second-best traditional DGSS method per text).
- On SYNTHIA to Cityscapes/BDD/Mapillary, CLOUDS with ConvNext-L attains 52.1% mIoU, outperforming FC-CLIP (35.6%) by a large margin.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CLOUDS with ConvNeXt-L instead of FC-CLIP | GTA5-trained semantic segmentation · GTA5 → {Cityscapes, BDD100K, Mapillary} | average mIoU: 57.0 -> 61.5 | Helped |
| Use CLOUDS with ConvNeXt-L instead of FC-CLIP | SYNTHIA-trained semantic segmentation · SYNTHIA → {Cityscapes, BDD100K, Mapillary} | average mIoU: 35.6 -> 52.1 | Helped |
| Use CLOUDS with ResNet-50 instead of PODA | Clear-weather source semantic segmentation · ACDC Night | mIoU: 25.0 -> 29.4 | Helped |
| Use CLOUDS with ResNet-50 instead of PODA | Clear-weather source semantic segmentation · ACDC Snow | mIoU: 43.9 -> 52.1 | Helped |
| Use CLOUDS with ResNet-50 instead of PODA | Clear-weather source semantic segmentation · ACDC Rain | mIoU: 42.3 -> 49.7 | Helped |
| Add LLM- and diffusion-generated data for self-training (ablation) | CLOUDS with ConvNeXt-L · GTA5 → {Cityscapes, BDD100K, Mapillary} | average mIoU: 58.5 -> 58.6 | No clear effect |
| Add LLM- and diffusion-generated data for self-training (ablation) | CLOUDS with ResNet-50 · GTA5 → {Cityscapes, BDD100K, Mapillary} | average mIoU: 50.0 -> 50.7 | Helped |
| Add SAM pseudo-label refinement (ablation) | CLOUDS with ConvNeXt-L · GTA5 → {Cityscapes, BDD100K, Mapillary} | average mIoU: 58.6 -> 61.5 | Helped |
| Add SAM pseudo-label refinement (ablation) | CLOUDS with ResNet-50 · GTA5 → {Cityscapes, BDD100K, Mapillary} | average mIoU: 50.7 -> 53.3 | Helped |
| Add SAM pseudo-label refinement (ablation) | CLOUDS with ResNet-101 · GTA5 → {Cityscapes, BDD100K, Mapillary} | average mIoU: 53.3 -> 54.7 | Helped |
| Increase generated dataset size from 100 to 5000 images (ablation) | CLOUDS with ConvNeXt-L · GTA5 → {Cityscapes, BDD100K, Mapillary} | mIoU: 58.2 -> 61.5 | Helped |
| Increase generated dataset size beyond 5000 images (ablation) | CLOUDS with ConvNeXt-L · GTA5 → {Cityscapes, BDD100K, Mapillary} | mIoU: 61.5 -> 61.6 | No clear effect |
Surprising findings
- Adding LLM and diffusion-generated data alone gives only marginal gains with ConvNext-L and ResNet-50; the major gain (~+3% mIoU) comes from SAM-based pseudo-label refinement.
- Fine-tuning the CLIP encoder degrades generalization (61.5% frozen vs 58.1% trainable average mIoU).
Limitations
- Gains from generated diffusion data are marginal with some backbones due to noisy pseudo labels from domain shift in generated images.
- Performance drops with more SAM prompt points, indicating sensitivity to pseudo-label noise.
- Comparison with PTDiffSeg is limited to Cityscapes since its code is unavailable.
Experimental setup
Datasets: GTA5 SYNTHIA Cityscapes BDD100K Mapillary ACDC LAION-5B ImageNet WIT
Benchmarks: GTA to Cityscapes/BDD100K/Mapillary SYNTHIA to Cityscapes/BDD100K/Mapillary ACDC
Metrics: mIoU
Baselines: IBN-Net ISW SHADE TLDR MoDify HRDA Grounding-SAM CAT-Seg ODISE FC-CLIP PTDiffSeg DRPC
Builds on: Mask2Former CLIP Segment Anything Model Stable Diffusion HRDA FC-CLIP
Closest papers by meaning
- Exploring Semantic Consistency and Style Diversity for Domain Generalized Semantic Segmentation (AAAI 2025 · similarity 0.90)
- Stronger Fewer & Superior: Harnessing Vision Foundation Models for Domain Generalized Semantic Segmentation (CVPR 2024 · similarity 0.90)
- MaskViM: Domain Generalized Semantic Segmentation with State Space Models (AAAI 2025 · similarity 0.89)
- Better to Teach than to Give: Domain Generalized Semantic Segmentation via Agent Queries with Diffusion Model Guidance (ICML 2025 · similarity 0.89)
- Sparse All-Layer Connector for Domain Generalised Semantic Segmentation (NeurIPS 2026 · similarity 0.89)
- A Simple Recipe for Language-guided Domain Generalized Segmentation (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.