Research map / Image generation / Text-to-image diffusion
Slight Corruption in Pre-training Data Makes Better Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies how slight condition corruption (label/text swaps) in pre-training data affects over 50 conditional diffusion models, proves via a Gaussian mixture analysis that it raises entropy and lowers 2-Wasserstein distance, and proposes conditional embedding perturbation (CEP) as a pre-training regularizer.
Problem
Web-crawled pre-training datasets for diffusion models contain corrupted image-condition pairs, yet the effect of such condition corruption on pre-training and downstream adaptation of diffusion models is largely unexplored.
Approach
Slight synthetic condition corruption during pre-training improves quality, diversity and fidelity of diffusion models; a GMM analysis explains this via entropy increase and reduced 2-Wasserstein distance, motivating CEP, which adds uniform or Gaussian noise to conditional embeddings on the fly.
Key results
- CEP improves LDM-4 on IN-1K FID from 9.44 to 6.91 and IS from 138.46 to 180.77 (CEP-G, s=2.0).
- CEP improves DiT-XL/2 on IN-1K FID from 6.76 to 5.51 (CEP-U) and IS from 179.67 to 189.94.
- Slight corruption (eta up to 7.5%) lowers FID and raises IS and CLIP score relative to clean pre-training across class- and text-conditional LDMs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add conditional embedding perturbation during pre-training | LDM-4 and DiT-XL/2 · IN-1K pre-training | FID: 9.44 -> 6.91; 6.76 -> 5.51 | Helped |
| Add conditional embedding perturbation during pre-training | LDM-4 and DiT-XL/2 · IN-1K pre-training | IS: 138.46 -> 180.77; 179.67 -> 189.94 | Helped |
| Add conditional embedding perturbation during pre-training | LDM-4 and LCM-v1.5 · CC3M pre-training | FID: 19.85 -> 17.93; 23.59 -> 22.91 | Helped |
| Add conditional embedding perturbation during pre-training | LDM-4 and LCM-v1.5 · CC3M pre-training | IS: 30.09 -> 30.77; 39.15 -> 40.31 | Helped |
| Add conditional embedding perturbation during pre-training | LDM-4 · IN-1K Canny ControlNet personalization | FID: 11.59 -> 11.46 | Helped |
| Add conditional embedding perturbation during pre-training | LDM-4 · IN-1K SAM ControlNet personalization | FID: 13.74 -> 12.95 | Helped |
| Add conditional embedding perturbation during pre-training | LDM-4 · CC3M Canny ControlNet personalization | FID: 40.65 -> 34.57 | Helped |
| Add conditional embedding perturbation during pre-training | LDM-4 · CC3M SAM ControlNet personalization | FID: 42.64 -> 35.02 | Helped |
| Use input perturbation instead of conditional embedding perturbation (ablation) | LDM-4, DiT-XL/2, and LCM-v1.5 · Pre-training benchmarks | Precision | Hurt |
| Use input perturbation instead of conditional embedding perturbation (ablation) | LDM-4 · IN-100 ControlNet personalization | FID: 11.59 -> 12.31; 13.74 -> 13.61; 40.65 -> 40.12; 42.64 -> 43.79 | Mixed |
| Increase conditional embedding perturbation magnitude (vary γ) (ablation) | Class-conditional LDM-4 · IN-1K pre-training | FID | Mixed |
| Use uniform rather than Gaussian conditional embedding perturbation (ablation) | LDM-4 and DiT-XL/2 · Pre-training benchmarks | FID: 6.91 vs 7.00; 5.51 vs 5.92; 17.93 vs 18.59; 22.91 vs 23.40 | Mixed |
Surprising findings
- Slight synthetic label or text corruption during pre-training improves generation quality and diversity over clean pre-training, including for downstream ControlNet and T2I-Adapter personalization.
- CEP achieves lower FID with larger embedding-space deviation from clean embeddings, i.e., more corruption in embedding space can help.
Limitations
- Due to limited computing resources, not all DM types could be studied on larger datasets.
- The theoretical analysis relies on GMM and linear-network assumptions.
- Evaluation of image generation quality remains an open question; existing metrics were used.
Experimental setup
Datasets: ImageNet-1K CC3M ImageNet-100 MS-COCO CIFAR-100
Benchmarks: ImageNet-1K MS-COCO
Metrics: FID IS Precision Recall CLIP score sFID TopPR F1 RMD entropy memorization ratio
Baselines: clean pre-trained DMs input perturbation (IP) Dropout Label Smoothing
Builds on: Latent Diffusion Models classifier-free guidance DiT Latent Consistency Models ControlNet
Closest papers by meaning
- Random Conditioning for Diffusion Model Compression with Distillation (CVPR 2025 · similarity 0.89)
- Stochastic Forward–Backward Deconvolution: Training Diffusion Models with Finite Noisy Datasets (ICML 2025 · similarity 0.89)
- On Error Propagation of Diffusion Models (ICLR 2024 · similarity 0.89)
- Consistent Diffusion Meets Tweedie: Training Exact Ambient Diffusion Models with Noisy Data (ICML 2024 · similarity 0.89)
- On improved Conditioning Mechanisms and Pre-training Strategies for Diffusion Models (NeurIPS 2024 · similarity 0.89)
- Sparsely Supervised Diffusion (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.