Research map / Detection and segmentation / Open-set and few-shot detection
SemFlow: Binding Semantic Segmentation and Image Synthesis via Rectified Flow
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SemFlow, a single rectified-flow ODE on Stable Diffusion latents that transports images to pseudo-mask latents for semantic segmentation and back for semantic image synthesis, with a finite perturbation to add diversity.
Problem
Diffusion-based segmentation produces random outputs and needs many inference steps, while semantic segmentation and semantic image synthesis are typically solved by separate models, leaving the pair of reverse tasks ununified.
Approach
Model segmentation and semantic image synthesis as reverse transport problems between image and pseudo-mask latent distributions using one rectified flow ODE with a time-symmetric objective, drop Gaussian noise input, and add bounded uniform perturbation to masks to enable multi-modal synthesis without changing semantic labels.
Key results
- SemFlow reaches 38.6 mIoU on COCO-Stuff segmentation with Euler-25 sampling, versus 16.1 (DDIM-200) and 20.2 (DDPM-200) for diffusion segmentation baselines.
- SemFlow achieves 32.6 FID and 0.393 LPIPS on CelebAMask-HQ synthesis with one model, better FID than SC-GAN (19.2 is better; SemFlow is worse than SC-GAN, pix2pixHD and SPADE on FID but better than pix2pixHD and SPADE on LPIPS) while trailing BBDM's FID of 21.4.
- With a single Euler step SemFlow still gets 28.3 mIoU on COCO-Stuff, while DSM-DDPM with 5 steps gets 12.3 mIoU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SemFlow instead of DSM-DDPM | COCO-Stuff semantic segmentation · COCO-Stuff | mIoU: 20.2 -> 38.6 | Helped |
| Use SemFlow instead of MaskFormer | COCO-Stuff semantic segmentation · COCO-Stuff | mIoU: 41.9 -> 38.6 | Hurt |
| Use SemFlow instead of BBDM | CelebAMask-HQ semantic image synthesis · CelebAMask-HQ | FID: 21.4 -> 32.6 | Hurt |
| Use SemFlow instead of SC-GAN | CelebAMask-HQ semantic image synthesis · CelebAMask-HQ | LPIPS: 0.395 -> 0.393 | Hurt |
| Add finite perturbation to pseudo masks (ablation) | SemFlow semantic image synthesis · CelebAMask-HQ | image detail and multimodal diversity | Helped |
| Reduce inference steps from 25 to 1 (ablation) | SemFlow semantic segmentation · COCO-Stuff | mIoU: 38.6 -> 28.3 | Hurt |
| Reduce inference steps from 25 to 5 (ablation) | SemFlow semantic segmentation · COCO-Stuff | mIoU: 38.6 -> 36.9 | Hurt |
| Use 1 inference step instead of 25 (ablation) | DSM-DDPM semantic segmentation · COCO-Stuff | mIoU: 5.3 -> 0.1 | Hurt |
| Use 1 inference step instead of 25 (ablation) | DSM-DDIM semantic segmentation · COCO-Stuff | mIoU: 4.0 -> 0.1 | Hurt |
Surprising findings
- SemFlow with one Euler step (28.3 mIoU) outperforms DSM baselines run with 200 steps (20.2 mIoU max).
Limitations
- Performance gap remains against specialist discriminative segmentation models such as MaskFormer (38.6 vs 41.9 mIoU).
- Trained on relatively small datasets and limited compute; authors note scaling data is left to future work.
- Network architecture and ODE solver design are not explored in depth.
Experimental setup
Datasets: COCO-Stuff-164k CelebAMask-HQ Cityscapes
Benchmarks: COCO-Stuff CelebAMask-HQ Cityscapes
Metrics: mIoU FID LPIPS
Baselines: MaskFormer DSM-DDIM DSM-DDPM pix2pixHD SPADE SC-GAN BBDM CycleGAN
Builds on: rectified flow Stable Diffusion DDPM DDIM
Closest papers by meaning
- $\mathbf{D^{3}S^{2}}$: Diffusion-Guided Dataset Distillation for Semantic Segmentation (NeurIPS 2026 · similarity 0.87)
- Seg4Diff: Unveiling Open-Vocabulary Semantic Segmentation in Text-to-Image Diffusion Transformers (NeurIPS 2025 · similarity 0.87)
- Rethinking Vector Field Learning for Generative Segmentation (NeurIPS 2026 · similarity 0.86)
- EmerDiff: Emerging Pixel-level Semantic Knowledge in Diffusion Models (ICLR 2024 · similarity 0.86)
- Scaling Rectified Flow Transformers for High-Resolution Image Synthesis (ICML 2024 · similarity 0.85)
- Rethinking Decoders for Transformer-based Semantic Segmentation: A Compression Perspective (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.