Research map / Image generation / Autoregressive and latent generation

UniGS: Unified Representation for Image Generation and Segmentation

CVPR 2024 · Image generation › Autoregressive and latent generation · approach: Classical generative image synthesis · 5 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes UniGS, a latent diffusion model (initialized from Stable Diffusion inpainting) that generates both images and entity-level segmentation masks encoded as colormaps, using a location-aware palette and a progressive dichotomy decoding module.

Problem

Unifying image generation and entity-level segmentation in one diffusion model, where varying entity counts and scarce segmentation training data make direct colormap-based mask generation difficult.

Approach

Represent entity masks as a colormap in the same latent space as images, assign colors by entity center-of-mass location (location-aware palette), decode colormaps to masks via depth-first binary k-means splitting without knowing the cluster count (progressive dichotomy module), and train under an inpainting protocol so partially labeled segmentation datasets can be used.

Key results

• Inpainting: UniGS1.5I reaches FID 3.78 (single object) vs 4.95 for SD1.5 inpainting, and 5.89 vs 7.82 on multiple objects.
• Image synthesis: UniGS achieves mIoU 0.781 and FID 15.272 on single-object synthesis vs 0.306 and 34.434 for T2I-Adapter.
• Referring segmentation: UniGS reaches mIoU 0.808 vs 0.815 for Mask2Former (Swin-Large), without explicit segmentation loss; entity segmentation lags CropFormer (0.631 vs 0.664 mIoU).

Measured effects reported in the paper

Surprising findings

• Image inpainting FID and CLIP scores improve over Stable Diffusion inpainting even when initialized from SD v1.4, attributed to the mask representation constraining generation.
• The progressive dichotomy module matches k-means with ground-truth cluster numbers (mIoU 0.808 vs 0.810), despite not knowing cluster counts.

Limitations

• Entity segmentation performance is notably below state-of-the-art (0.631 mIoU vs 0.664 for CropFormer).
• Multi-task unified model underperforms single-task models at 48 epochs on several tasks, and only marginally improves at 200 epochs.
• Agglomerative clustering runs out of memory, and DBSCAN performs poorly, so the decoding design relies on the chosen dichotomy procedure.

Experimental setup

Datasets: COCO Open Images EntitySeg ADE20K

Benchmarks: COCO validation (1000 sampled images) EntitySeg

Metrics: FID CLIP score mIoU recall AP (entity, non-overlapped)

Baselines: Stable Diffusion inpainting ControlNet T2I-Adapter Mask2Former SAM CropFormer ConInst-Entity K-Means DBSCAN Agglomerative clustering

Builds on: Stable Diffusion Painter InstructDiffusion latent diffusion models

Closest papers by meaning

• Factorized Diffusion Architectures for Unsupervised Image Generation and Segmentation (NeurIPS 2024 · similarity 0.86)
• MaskFactory: Towards High-quality Synthetic Data Generation for Dichotomous Image Segmentation (NeurIPS 2024 · similarity 0.86)
• High-Precision Dichotomous Image Segmentation via Probing Diffusion Capacity (ICLR 2025 · similarity 0.86)
• Rethinking Vector Field Learning for Generative Segmentation (NeurIPS 2026 · similarity 0.86)
• $\mathbf{D^{3}S^{2}}$: Diffusion-Guided Dataset Distillation for Semantic Segmentation (NeurIPS 2026 · similarity 0.85)
• Explore In-Context Segmentation via Latent Diffusion Models (AAAI 2025 · similarity 0.85)