Research map / Label-efficient and robust learning / Domain adaptation and generalization
DynAlign: Unsupervised Dynamic Taxonomy Alignment for Cross-Domain Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DynAlign, a training-free-at-inference framework that combines a UDA segmenter (HRDA) with LLM-based taxonomy mapping, SAM mask proposals, and CLIP (ConvCLIP) multi-scale region reclassification to segment GTA images into new fine-grained target taxonomies (Mapillary Vistas, IDD) without target annotations.
Problem
Unsupervised domain adaptation for semantic segmentation usually assumes identical source and target label spaces, ignoring label-level taxonomy gaps (finer-grained or novel classes) between domains; existing taxonomy-adaptive DA methods need few-shot target labels.
Approach
Combine in-domain UDA predictions (source label space) with foundation-model priors: an LLM maps each source label to candidate target labels with context descriptions, SAM produces mask proposals, and CLIP with multi-scale visual features reassigns each mask to a target label via text-image similarity, fusing domain and prior knowledge without target annotations.
Key results
- On GTA to Mapillary Vistas, DynAlign reaches 36.7% mIoU and 53.0% mAcc, versus 32.9% mIoU for HRDA + Grounded-SAM.
- On GTA to IDD, DynAlign reaches 41.7% mIoU and 57.7% mAcc, versus 40.2% mIoU for HRDA + OWL-ViT-SAM.
- Pseudo-label training of a Mask2Former on Mapillary raises mIoU from 36.7% to 38.9% and mAcc from 53.0% to 56.2%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DynAlign instead of HRDA + Grounded-SAM | Mapillary Vistas segmentation · Mapillary Vistas | mIoU (all classes): 32.9 -> 36.7 | Helped |
| Use DynAlign instead of HRDA + OwlVIT-SAM | IDD segmentation · IDD | mIoU (all classes): 40.2 -> 41.7 | Helped |
| Train a segmentation model on DynAlign-generated pseudo-labels | Mapillary Vistas segmentation · Mapillary Vistas val | mIoU: 36.7% to 38.9% | Helped |
| Add multi-scale visual features without context-aware text features (ablation) | DynAlign · Mapillary Vistas | mIoU: 29.0% to 35.6% | Helped |
| Add multi-scale visual features without context-aware text features (ablation) | DynAlign · IDD | mIoU: 37.9% to 39.9% | Helped |
| Add context-aware text features without multi-scale vision (ablation) | DynAlign · Mapillary Vistas | mIoU: 29.0% to 29.4% | Helped |
| Add context-aware text features without multi-scale vision (ablation) | DynAlign · IDD | mIoU: 37.9% to 38.0% | No clear effect |
| Combine multi-scale visual and context-aware text features (ablation) | DynAlign · Mapillary Vistas | mIoU: 29.0% to 36.7% | Helped |
| Replace ConvCLIP with CLIP, MaskCLIP, or SCLIP (ablation) | DynAlign · Mapillary Vistas | mIoU: 36.7; 35.0; 34.7; 36.0 | Hurt |
| Replace ConvCLIP with CLIP, MaskCLIP, or SCLIP (ablation) | DynAlign · IDD | mIoU: 41.7; 38.0; 37.5; 39.4 | Hurt |
| Replace HRDA with MIC (ablation) | DynAlign · Mapillary Vistas | mIoU: 36.7 -> 37.7 | Helped |
| Replace SAM with MobileSAM (ablation) | DynAlign · Mapillary Vistas | mIoU: 36.7 -> 35.2 | Hurt |
Surprising findings
- Pure open-vocabulary methods (Grounded-SAM, OWL-ViT-SAM) perform poorly on street scenes, and naive combination with HRDA leaves a gap to DynAlign mainly on unknown classes.
Limitations
- Full inference pipeline is slow (about 64 s per image) and uses about 1.0B total parameters across HRDA, SAM and CLIP.
- Taxonomy mappings were generated by GPT-4 and manually refined, so the method relies on human intervention per dataset pair.
- Performance on some unknown classes is weak (e.g., mIoU 19.6% on Mapillary unknown classes).
Experimental setup
Datasets: GTA Mapillary Vistas IDD Cityscapes Synthia
Benchmarks: GTA to Mapillary Vistas GTA to IDD GTA to Cityscapes Synthia to Cityscapes
Metrics: mIoU mAcc
Baselines: Grounded-SAM OWL-ViT-SAM HRDA + Grounded-SAM HRDA + OWL-ViT-SAM MIC MobileSAM
Builds on: HRDA DAFormer ConvCLIP Segment Anything
Closest papers by meaning
- Universal Domain Adaptation for Semantic Segmentation (CVPR 2025 · similarity 0.90)
- Balanced Learning for Domain Adaptive Semantic Segmentation (ICML 2025 · similarity 0.89)
- Exploring Semantic Consistency and Style Diversity for Domain Generalized Semantic Segmentation (AAAI 2025 · similarity 0.88)
- Target Semantics Clustering via Text Representations for Robust Universal Domain Adaptation (AAAI 2025 · similarity 0.87)
- Adapter Naturally Serves as Decoupler for Cross-Domain Few-Shot Semantic Segmentation (ICML 2025 · similarity 0.87)
- UniDSeg: Unified Cross-Domain 3D Semantic Segmentation via Visual Foundation Models Prior (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.