Research map / Image generation / Fast diffusion sampling
PaGoDA: Progressive Growing of a One-Step Generator from a Low-Resolution Diffusion Teacher
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PaGoDA, a three-stage pipeline (train a diffusion model on downsampled data, distill it into a one-step generator via DDIM-inversion reconstruction plus adversarial loss, then progressively grow a super-resolution decoder) that reaches SOTA one-step FIDs on ImageNet 64-512 and competitive text-to-image results.
Problem
Training diffusion models at high resolution is computationally expensive, and one-step distilled generators typically inherit the teacher's quality and do not cover the prior well, making high-resolution one-step generation costly and unstable.
Approach
Train the diffusion teacher at low resolution, distill it with a DDIM-inversion-based reconstruction loss (data-to-noise) plus an adversarial loss, then progressively add ResNet upscaling blocks with frozen lower-resolution layers to reach high resolution; a classifier-free guided adversarial loss and CFG weight estimator extend this to text-to-image.
Key results
- PaGoDA with one step achieves FID 1.21 at 64x64 and 1.80 at 512x512 on ImageNet without CFG, outperforming listed baselines at each resolution.
- Training the diffusion teacher on 8x downsampled data is claimed to reduce training cost by 64x.
- On text-to-image (MSCOCO-2014 FID-30K), PaGoDA reaches 10.23 FID at 0.05s per image, versus 9.62 for SD1.5 at 2.9s.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| distill the teacher into a one-step PaGoDA generator | ImageNet 64×64 generation from EDM · ImageNet 64×64 | FID: 2.44 -> 1.21 | Helped |
| use a 32×32 rather than 64×64 base resolution (ablation) | PaGoDA · ImageNet 128×128 | FID: 1.48 -> 1.61 | Hurt |
| use a 32×32 rather than 64×64 base resolution (ablation) | PaGoDA · ImageNet 256×256 | FID: 1.56 -> 1.83 | Hurt |
| add adversarial loss during super-resolution | PaGoDA Stage 3 · ImageNet super-resolution | upscaling performance | Helped |
| add reconstruction loss during super-resolution | PaGoDA Stage 3 · ImageNet super-resolution | object alignment across resolution jumps | Helped |
| train using only 1% of data-latent pairs for the reconstruction loss (ablation) | PaGoDA · ImageNet generation | sample quality and diversity: 1% | No clear effect |
| use recaptioned rather than original text prompts (ablation) | PaGoDA text-to-image · Text-to-image generation | FID and CLIP metrics | Helped |
| add CLIP regularization (ablation) | PaGoDA text-to-image · Text-to-image generation | FID and CLIP scores | Helped |
| use unconditional rather than conditional discriminators (ablation) | PaGoDA ImageNet training · ImageNet generation | performance | No clear effect |
| use LPIPS features for reconstruction (ablation) | PaGoDA ImageNet training · ImageNet generation | performance | No clear effect |
Surprising findings
- The one-step generator distilled in Stage 2 consistently outperforms its own multi-step teacher diffusion model.
- Upscaling 64x64 to 512x512 yields only a small FID rise (1.21 to 1.80), similar to EDM2's latent-diffusion gap, suggesting pixel-space upscaling is competitive with latent diffusion.
Limitations
- Text-to-image Stage 1 pretraining was not performed due to concerns about sensitive content in LAION, so the work relies on the pretrained DeepFloyd-IF teacher.
- Error bars are not reported due to limited computational resources.
- The paper notes that comparisons with other upscaling methods (SD, CDM) are not direct and that the framework is not yet applied in latent space.
Experimental setup
Datasets: ImageNet CC12M COYO-700M MSCOCO-2014 MSCOCO-2017 LAION-2B YFCC100M
Benchmarks: ImageNet 64x64 ImageNet 128x128 ImageNet 256x256 ImageNet 512x512 MSCOCO-2014 MSCOCO-2017
Metrics: FID Inception Score recall CLIP score speed
Baselines: StyleGAN-XL CTM VDM++ simple Diffusion RIN DiT-XL EDM2-XXL SD1.5 DeepFloyd-IF LCM UFOGen ADD
Builds on: EDM StyleGAN-XL consistency trajectory models progressive growing of GANs DDIM
Closest papers by meaning
- Masked Autoencoders Are Effective Tokenizers for Diffusion Models (ICML 2025 · similarity 0.90)
- DiffuseHigh: Training-Free Progressive High-Resolution Image Synthesis Through Structure Guidance (AAAI 2025 · similarity 0.90)
- Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference (NeurIPS 2024 · similarity 0.89)
- Efficient Dataset Distillation via Minimax Diffusion (CVPR 2024 · similarity 0.89)
- On improved Conditioning Mechanisms and Pre-training Strategies for Diffusion Models (NeurIPS 2024 · similarity 0.89)
- Plug-and-Play Diffusion Distillation (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.