Research map / Image generation / Fast diffusion sampling
A Simple Latent Diffusion Approach for Panoptic Segmentation and Mask Inpainting
Wouter Van Gansbeke, Bert De Brabandere
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace shallow autoencoder (2M params) with finetuned Stable Diffusion VAE (84M) (ablation) | LDMSeg · COCO val2017 (class-agnostic panoptic) | PQ: 50.8 -> 50.9 | No clear effect |
| add KL loss with weight above 1e-5 to the autoencoder (ablation) | LDMSeg autoencoder · COCO val2017 | reconstruction quality | Hurt |
| use bit encoding instead of color encoding for instance IDs (ablation) | LDMSeg autoencoder · COCO autoencoder reconstruction | PQ: 89.1 -> 89.9 | Improved |
| use bit encoding instead of positional encoding (ablation) | LDMSeg autoencoder · COCO autoencoder reconstruction | PQ: 88.2 -> 89.9 | Improved |
| increase DDIM inference steps from 1 to 50 (ablation) | LDMSeg · COCO class-agnostic panoptic, 50k iters | PQ: 8.4 -> 51.9 | Improved |
| increase sampling steps beyond 50 (ablation) | LDMSeg · COCO class-agnostic panoptic, 50k iters | PQ: 51.9 -> 52.2 | No clear effect |
| swap Stable Diffusion VAE image encoder for ViT-B/14 DINOv2 weights (ablation) | LDMSeg · COCO panoptic component analysis | PQ: 40.6 -> 43.7 | Improved |
| use DDPM scheduler instead of DDIM (ablation) | LDMSeg · COCO panoptic component analysis | PQ: 43.7 -> 44.3 | Improved |
| add EMA of model weights (ablation) | LDMSeg · COCO panoptic component analysis | PQ: 40.3 -> 40.6 | Improved |
| train jointly on semantic, panoptic and depth with task embeddings versus single-task | LDMSeg · COCO val2017 multi-task | Semantic seg mIoU: 61.5 -> 60.1 | Hurt |
| train jointly on three tasks versus single-task | LDMSeg · COCO val2017 multi-task | Panoptic PQ: 44.3 -> 44.1 | No clear effect |
| add CLIP image embeddings or BLIP captions via cross-attention (ablation) | LDMSeg · COCO class-agnostic | sample quality | No clear effect |
Closest papers by meaning
- A Generalist Framework for Panoptic Segmentation of Images and Videos (ICCV 2023 · similarity 0.91)
- Explore In-Context Segmentation via Latent Diffusion Models (AAAI 2025 · similarity 0.89)
- Conditional Latent Diffusion Models for Zero-Shot Instance Segmentation (ICCV 2025 · similarity 0.88)
- Real-Time Panoptic Segmentation From Dense Detections (CVPR 2020 · similarity 0.88)
- Scene-Centric Unsupervised Panoptic Segmentation (CVPR 2025 · similarity 0.86)
- Using Diffusion Priors for Video Amodal Segmentation (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
