Research map / Image generation / Text-to-image diffusion
Unsupervised Modality Adaptation with Text-to-Image Diffusion Models for Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends unsupervised domain adaptation for semantic segmentation from image-to-image to image-to-other-modality (depth, infrared, event) by using a Stable Diffusion backbone with diffusion-based pseudo-label generation and label-palette latent regression.
Problem
Unsupervised adaptation for semantic segmentation from a labeled RGB source to unlabeled target visual modalities (depth, infrared, event), where large modality gaps make pseudo-labels unreliable and latent diffusion features are low-resolution.
Approach
Use a pre-trained text-to-image diffusion model (Stable Diffusion v1-4) as the self-training backbone; add annealed latent noise to target inputs to stabilize pseudo-labels (DPLG), and convert one-hot labels to RGB via a palette and regress their VAE latents so the frozen-architecture VAE decoder yields high-resolution features (LPLR).
Key results
- MADM reaches 53.49% MIoU on Cityscapes to DELIVER-Depth, 62.23% on FMB-Infrared, and 56.31% on DSEC-Event, outperforming Rein by +3.26, +1.58, and +4.45 points.
- Ablation on the depth/infrared/event modalities: DPLG adds +2.40 average MIoU and LPLR adds +1.85; combined gives +4.09 over baseline (53.05 average).
- Distilling MADM into a DAFormer (85M params) yields 54.03% MIoU on DSEC-Event at 1.3 extra training hours.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MADM instead of Rein | Cityscapes-Image → DELIVER-Depth · DELIVER-Depth | MIoU: 50.23 -> 53.49; +3.26% | Helped |
| use MADM instead of Rein | Cityscapes-Image → FMB-Infrared · FMB-Infrared | MIoU: 60.65 -> 62.23; +1.58% | Helped |
| use MADM instead of Rein | Cityscapes-Image → DSEC-Event · DSEC-Event | MIoU: 51.86 -> 56.31; +4.45% | Helped |
| add DPLG without LPLR (ablation) | MADM · Depth | MIoU: 50.61 -> 51.65 | Helped |
| add DPLG without LPLR (ablation) | MADM · Infrared | MIoU: 56.28 -> 61.86 | Helped |
| add DPLG without LPLR (ablation) | MADM · Event | MIoU: 52.27 -> 52.84 | Helped |
| add LPLR without DPLG (ablation) | MADM · Depth | MIoU: 50.61 -> 52.91 | Helped |
| add LPLR without DPLG (ablation) | MADM · Infrared | MIoU: 56.28 -> 58.75 | Helped |
| add LPLR without DPLG (ablation) | MADM · Event | MIoU: 52.27 -> 53.05 | Helped |
| add both DPLG and LPLR (ablation) | MADM baseline · Depth, Infrared, and Event | MIoU: +4.09% average | Helped |
| increase β from 40 to 60 with γ=5,000 (ablation) | DPLG · Depth | MIoU: 52.46 -> 53.49 | Helped |
| increase β from 60 to 80 with γ=5,000 (ablation) | DPLG · Depth | MIoU: 53.49 -> 52.85 | Hurt |
Surprising findings
- Noise-free single-step diffusion features are not optimal for pseudo-labeling; moderate latent noise (k=10 to 50) improves segmentation while excessive noise (k=200) degrades it.
- The baseline built on the diffusion backbone without DPLG or LPLR already performs on par with prior SoTA methods, attributed to the generalization of text-to-image diffusion models.
Limitations
- Computation is far higher than existing UDASS networks (949M params, 1.38 s/iter).
- Relies on a single source modality (Cityscapes RGB) and three target modalities; generality to other modalities not shown.
- Class merging is needed because label sets differ across datasets.
Experimental setup
Datasets: Cityscapes DELIVER-Depth FMB-Infrared DSEC-Event LAION-5B
Benchmarks: DELIVER-Depth FMB-Infrared DSEC-Event
Metrics: MIoU
Baselines: DAFormer MIC PiPa Rein
Builds on: DAFormer MIC Stable Diffusion Rein Marigold
Closest papers by meaning
- Dual Diffusion for Unified Image Generation and Understanding (CVPR 2025 · similarity 0.88)
- Keep the Balance: A Parameter-Efficient Symmetrical Framework for RGB+X Semantic Segmentation (CVPR 2025 · similarity 0.88)
- Diffusion Bridge: Leveraging Diffusion Model to Reduce the Modality Gap Between Text and Vision for Zero-Shot Image Captioning (CVPR 2025 · similarity 0.87)
- Towards General Modality Translation with Contrastive and Predictive Latent Diffusion Bridge (NeurIPS 2025 · similarity 0.87)
- Seg4Diff: Unveiling Open-Vocabulary Semantic Segmentation in Text-to-Image Diffusion Transformers (NeurIPS 2025 · similarity 0.87)
- VidSeg: Training-free Video Semantic Segmentation based on Diffusion Models (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.