Research map / Image generation / Fast diffusion sampling
Training Diffusion-based Generative Models with Limited Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LD-Diffusion, an EDM-based latent diffusion model with a VAE-style compressing model, mixed augmentation with fixed probability (MAFP), patch training and OOD regularization, to train diffusion models from scratch on hundreds of images.
Problem
Diffusion models need large datasets; training them from scratch with limited data (hundreds of images) yields poor denoising score matching and low-quality samples.
Approach
A theoretical decomposition of total denoising score matching error into denoiser hypothesis-space approximation and estimation error, motivating a compressed latent denoiser space (pre-trained encoder/decoder from SD VAEs) plus a fixed-probability mix of Diff-Augment and ADA augmentations that avoids leaking, with OOD regularization for small-reference cases.
Key results
- On FFHQ-100 (256x256), LD-Diffusion reaches FID 28.51 versus 50.73 for EDM + DA, and FFHQ-2K FID 14.36 versus 27.17.
- On low-shot datasets LD-Diffusion reaches FID 13.00 on 100-shot Obama versus 37.10 for EDM + DA, and 4.70 on Grumpy cat versus 10.81.
- Ablation: removing the compressing model raises FFHQ-100 FID from 28.51 to 49.29, and removing MAFP raises it to 79.10.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LD-Diffusion instead of LPDM-8 | Diffusion models trained on FFHQ · FFHQ-100 | FID: 32.78 -> 28.51 | Helped |
| Use LD-Diffusion instead of LPDM-8 | Diffusion models trained on FFHQ · FFHQ-1K | FID: 19.67 -> 17.76 | Helped |
| Use LD-Diffusion instead of LPDM-8 | Diffusion models trained on FFHQ · FFHQ-2K | FID: 15.47 -> 14.36 | Helped |
| Use LD-Diffusion instead of LPDM-8 | Diffusion models trained on low-shot datasets · 100-shot Obama | FID: 14.27 -> 13.00 | Helped |
| Use LD-Diffusion instead of LPDM-8 | Diffusion models trained on low-shot datasets · 100-shot Animal-Face Panda | FID: 14.56 -> 13.31 | Helped |
| Use LD-Diffusion instead of LPDM-8 | Diffusion models trained on low-shot datasets · 100-shot Animal-Face Grumpy | FID: 5.13 -> 4.70 | Helped |
| Use LD-Diffusion instead of LPDM-8 | Diffusion models trained on low-shot datasets · 100-shot Animal-Face Cat | FID: 14.92 -> 12.77 | Helped |
| Use LD-Diffusion instead of LPDM-8 | Diffusion models trained on low-shot datasets · 100-shot Animal-Face Dog | FID: 15.95 -> 12.48 | Helped |
| Add patch training (ablation) | LD-Diffusion · FFHQ-100 | FID: 31.57 -> 28.51 | Helped |
| Add the compressing model (ablation) | LD-Diffusion · FFHQ-100 | FID: 49.29 -> 28.51 | Helped |
| Add MAFP (ablation) | LD-Diffusion · FFHQ-100 | FID: 79.10 -> 28.51 | Helped |
| Add OOD regularization (ablation) | LD-Diffusion · 100-shot Obama | FID: 13.28 -> 13.00 | Helped |
Surprising findings
- EDM2, which scales up EDM, performs worse than EDM on both limited-data settings (e.g., 112.26 vs 51.30 on 100-shot Obama).
Limitations
- Experiments are limited to image datasets, mainly FFHQ and low-shot benchmarks at 256x256 resolution.
- Relies on pre-trained Stable Diffusion VAE encoder/decoder, so the compression step depends on external pre-trained models.
- The theoretical insight is motivational and the paper does not give a formal bound on the benefit of the compressing model.
Experimental setup
Datasets: FFHQ FFHQ-100 FFHQ-1K FFHQ-2K FFHQ-140K 100-shot Obama 100-shot Grumpy cat 100-shot Panda AnimalFace-cat AnimalFace-dog
Benchmarks: FFHQ 256x256 low-shot datasets (100-shot Obama, Grumpy cat, Panda, AnimalFace cat/dog)
Metrics: FID Precision Recall
Baselines: EDM EDM + DA Patch Diffusion LPDM-8 EDM2
Builds on: EDM Patch Diffusion ADA DiffAugment Latent diffusion models
Closest papers by meaning
- Denoising Diffusion Bridge Models (ICLR 2024 · similarity 0.91)
- Evaluating the design space of diffusion-based generative models (NeurIPS 2024 · similarity 0.91)
- O(d/T) Convergence Theory for Diffusion Probabilistic Models under Minimal Assumptions (ICLR 2025 · similarity 0.91)
- Nearest Neighbour Score Estimators for Diffusion Generative Models (ICML 2024 · similarity 0.90)
- One-Step Diffusion Distillation through Score Implicit Matching (NeurIPS 2024 · similarity 0.90)
- What’s the score? Automated Denoising Score Matching for Nonlinear Diffusions (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.