Research map / Vision-language and multimodal / CLIP and zero-shot
A Simple Recipe for Language-guided Domain Generalized Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FAMix, which fine-tunes only the last layers of a CLIP-initialized DeepLabv3+ and mixes locally mined, language-driven class-wise style statistics with source styles to improve domain-generalized semantic segmentation.
Problem
Domain generalization for semantic segmentation: networks degrade on unseen domains, and naively using CLIP pretraining for segmentation fine-tuning overfits the source and performs worse than ImageNet pretraining on out-of-distribution data.
Approach
Freeze most of a CLIP image backbone, mine class-specific local styles with prompt-driven instance normalization using random style prompts concatenated with class names, and apply patch-wise convex mixing of source and mined style statistics (a class-wise, local MixStyle variant) during training with only cross-entropy loss.
Key results
- Single-source GTAV-trained ResNet-101 FAMix achieves 41.11 mean mIoU across unseen domains, versus 35.13 for SHADE (full data) in the same setting.
- With ResNet-50 trained on GTAV, FAMix reaches 38.88 mean mIoU, above TLDR (36.81) and WildNet (33.84).
- Multi-source (G+S) ResNet-50 FAMix reaches 48.84 mean mIoU versus 46.40 for TLDR.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FAMix instead of ImageNet-pretrained DeepLabv3+ | DeepLabv3+ with ResNet-50, trained on GTAV · OOD mean across Cityscapes, BDD-100K, Mapillary, SYNTHIA, and ACDC | mIoU (%): 25.90 -> 14.69 | Helped |
| Add minimal fine-tuning (Freeze) (ablation) | FAMix on GTAV · OOD mean across target datasets | mIoU (%): 14.69 -> 32.44 | Helped |
| Add language-driven style augmentation (ablation) | FAMix with minimal fine-tuning on GTAV · OOD mean across target datasets | mIoU (%): 32.44 -> 32.93 | Helped |
| Add local style mixing (ablation) | FAMix with minimal fine-tuning on GTAV · OOD mean across target datasets | mIoU (%): 32.44 -> 36.29 | Helped |
| Combine Freeze, Augment, and Mix (ablation) | FAMix on GTAV · OOD mean across target datasets | mIoU (%): 32.44 -> 38.88 | Helped |
| Use a random style prompt plus class name rather than a random character prompt plus class name (ablation) | FAMix prompt construction on GTAV · OOD mean across target datasets | mIoU (%): 37.29 -> 38.88 | Helped |
| Add the class name to the random style prompt (ablation) | FAMix prompt construction on GTAV · OOD mean across target datasets | mIoU (%): 37.35 -> 38.88 | Helped |
| Use local class-wise style mining instead of global style mining (ablation) | FAMix on GTAV · Cityscapes | mIoU (%): 45.51 -> 48.15 | Helped |
| Sample augmented styles from T rather than source styles S (ablation) | FAMix mixing on GTAV · OOD mean across target datasets | mIoU (%): 36.29 -> 38.88 | Helped |
| Replace language-driven augmentation with noise perturbation (ablation) | FAMix on GTAV · OOD mean across target datasets | mIoU (%): 38.88 -> 37.30 | Hurt |
| Use FAMix instead of vanilla MixStyle (ablation) | GTAV-trained segmentation model · OOD mean across target datasets | mIoU (%): 35.25 -> 38.88 | Helped |
| Use decoder-probing fine-tuning (DP-FT) instead of decoder probing (DP) | CLIP ResNet-50 segmentation model trained on GTAV · OOD mean across target datasets | mIoU (%): 29.99 -> 21.07 | Hurt |
Surprising findings
- Naive fine-tuning of CLIP-initialized segmentation networks performs worse than ImageNet pretraining on OOD data (14.69 vs 25.90 mean mIoU), and decoder-probing then fine-tuning falls behind decoder-probing alone.
- Replacing language-driven style mining with random character prompts barely hurts performance, suggesting the mixing itself drives much of the randomization.
Limitations
- Gains on some ACDC subsets and Cityscapes-source settings are less consistent; competitive rather than best on G and M when trained on Cityscapes.
- Sensitivity to the choice of SNR in the noise-based alternative and to the freezing depth requires per-setting tuning.
- Relies on the dominant class from ground-truth labels for style mining, so it depends on source annotations.
Experimental setup
Datasets: GTAV SYNTHIA Cityscapes BDD-100K Mapillary ACDC
Benchmarks: Cityscapes BDD-100K Mapillary SYNTHIA ACDC GTAV
Metrics: mIoU
Baselines: RobustNet SAN & SAW Pin the memory SHADE SiamDoGe DPCL SPC-Net NP WildNet TLDR MixStyle DP-FT
Builds on: MixStyle PØDA prompt-driven instance normalization CLIP DeepLabv3+ AdaIN
Closest papers by meaning
- Exploring Semantic Consistency and Style Diversity for Domain Generalized Semantic Segmentation (AAAI 2025 · similarity 0.89)
- CLIPCEIL: Domain Generalization through CLIP via Channel rEfinement and Image-text aLignment (NeurIPS 2024 · similarity 0.89)
- Stronger Fewer & Superior: Harnessing Vision Foundation Models for Domain Generalized Semantic Segmentation (CVPR 2024 · similarity 0.89)
- Collaborating Foundation Models for Domain Generalized Semantic Segmentation (CVPR 2024 · similarity 0.89)
- When and How Does CLIP Enable Domain and Compositional Generalization? (ICML 2025 · similarity 0.88)
- Generalize or Detect? Towards Robust Semantic Segmentation Under Multiple Distribution Shifts (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.