Research map / Image generation / Autoregressive and latent generation
Unlocking Pre-trained Image Backbones for Semantic Image Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DP-SIMS, a GAN for semantic image synthesis that uses a frozen ImageNet-pretrained backbone as the encoder of a UNet discriminator plus a cross-attention noise-injection generator, achieving SOTA FID/mIoU at ~100x faster inference than diffusion models.
Problem
Semantic image synthesis (label map to image) with GANs suffers from lower image quality on large diverse datasets, while diffusion models are slow at inference and less consistent with input label maps.
Approach
Use a frozen discriminatively pre-trained backbone (e.g., ConvNeXt-L) as the encoder of a UNet discriminator, with a sigmoid+BatchNorm feature conditioning step to balance backbone features, and a generator that injects noise via cross-attention and uses self-attention for context, plus focal, contrastive and diversity losses.
Key results
- On COCO-Stuff, DP-SIMS reaches FID 13.6 (vs 14.4 FLIS) and mIoUMF 65.2 (vs 52.1 OASIS) with ConvNeXt-L.
- On Cityscapes, DP-SIMS achieves FID 38.2 and mIoUMF 78.5, beating SDM's FID 42.1.
- Inference takes 0.04 s per 256x256 image versus 260 s for SDM and 17.4 s for PITI.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DP-SIMS instead of FLIS | Semantic image synthesis on COCO-Stuff · COCO-Stuff | FID: 14.4 -> 13.6 | Helped |
| Use DP-SIMS instead of OASIS | Semantic image synthesis on COCO-Stuff · COCO-Stuff | mIoUMF: 52.1 -> 65.2 | Helped |
| Use DP-SIMS instead of ECGAN++ | Semantic image synthesis on COCO-Stuff · COCO-Stuff | mIoU: 47.9 -> 57.7 | Helped |
| Use DP-SIMS instead of ECGAN++ | Semantic image synthesis on ADE-20K · ADE-20K | FID: 24.7 -> 22.7 | Helped |
| Use DP-SIMS instead of OASIS | Semantic image synthesis on ADE-20K · ADE-20K | mIoUMF: 53.5 -> 67.8 | Helped |
| Use DP-SIMS instead of SDM | Semantic image synthesis on Cityscapes · Cityscapes | FID: 42.1 -> 38.2 | Helped |
| Pre-train the discriminator backbone rather than use random initialization (ablation) | DP-SIMS on ADE-20K · ADE-20K | FID: 52.9 -> 22.7 | Helped |
| Pre-train on ImageNet-21k at 384 resolution rather than ImageNet-21k at 224 resolution (ablation) | DP-SIMS on ADE-20K · ADE-20K | mIoUMF: 64.1 -> 67.8 | Helped |
| Add sigmoid feature normalization (ablation) | DP-SIMS discriminator on ADE-20K · ADE-20K | FID: 23.3 -> 22.7 | Helped |
| Add BatchNorm feature normalization (ablation) | DP-SIMS discriminator on ADE-20K · ADE-20K | mIoUMF: 64.0 -> 67.8 | Helped |
| Add self-attention layers (ablation) | DP-SIMS generator on ADE-20K · ADE-20K | FID: 23.7 -> 22.7 | Helped |
| Use cross-attention noise injection rather than concatenation (ablation) | DP-SIMS generator on ADE-20K · ADE-20K | mIoUMF: 64.5 -> 67.8 | Helped |
Surprising findings
- Swin transformer backbones, despite best ImageNet accuracy, performed worse and were unstable as discriminator backbones.
- Adding BatchNorm to Projected GAN's feature conditioning degraded performance.
Limitations
- Swin backbones are unstable as discriminator encoders and need gradient clipping
- Diffusion-based comparisons rely on numbers from original papers for some metrics
- FID can be biased when pre-training dataset is the same as the Inception model dataset (ImageNet)
Experimental setup
Datasets: COCO-Stuff ADE-20K Cityscapes ImageNet-1k ImageNet-21k
Benchmarks: COCO-Stuff ADE-20K Cityscapes
Metrics: FID mIoU mIoUMF LPIPS inference time
Baselines: OASIS SDM PITI FLIS ECGAN++ DP-GAN PoE-GAN SPADE Pix2pixHD
Builds on: Projected GAN OASIS DP-GAN SPADE InfoNCE
Closest papers by meaning
- Learning to Predict Layout-to-image Conditional Convolutions for Semantic Image Synthesis (NeurIPS 2019 · similarity 0.93)
- You Only Need Adversarial Supervision for Semantic Image Synthesis (ICLR 2021 · similarity 0.92)
- Linear Semantics in Generative Adversarial Networks (CVPR 2021 · similarity 0.92)
- Semantic Pyramid for Image Generation (CVPR 2020 · similarity 0.90)
- Edge Guided GANs with Contrastive Learning for Semantic Image Synthesis (ICLR 2023 · similarity 0.90)
- Boosting Generative Image Modeling via Joint Image-Feature Synthesis (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.