Research map / Adversarial robustness and security / Attacks on text and graphs
Improved Diffusion-based Generative Model with Better Adversarial Robustness
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes efficient adversarial training (Free-AT style) for diffusion probabilistic models and consistency models, framed as DRO against distribution mismatch between training and sampling, improving FID on CIFAR-10, ImageNet 64, and MS-COCO text-to-image.
Problem
Diffusion models suffer from a distribution mismatch (exposure bias) between training inputs (ground-truth noisy data) and inference inputs (model-generated intermediate states), degrading sample quality, especially with few sampling steps; consistency models inherit this issue.
Approach
Shows that DRO over the intermediate-state distribution is equivalent to adversarial training with a perturbation on the noisy input (for DPM) and on the ODE-solver estimate (for CM); uses an efficient single-loop AT (Free-AT style) that adds little cost over standard training.
Key results
- ADM-AT improves DPM-Solver FID on ImageNet 64 at 5 NFEs from 27.72 (original) to 17.36.
- On CIFAR-10 with DDIM at 5 NFEs, ADM-AT reaches FID 26.38 versus 34.28 for the original ADM.
- LCM-AT on MS-COCO 512x512 improves 1-step FID from 25.43 to 23.34 and 4-step FID from 11.61 to 10.31, with comparable CLIP scores.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply adversarial training to ADM | ADM · CIFAR-10 32x32, DPM-Solver, 5 NFEs | FID: 22.98 -> 18.40 | Helped |
| Apply adversarial training to ADM | ADM · CIFAR-10 32x32, DPM-Solver, 50 NFEs | FID: 3.12 -> 3.01 | Helped |
| Apply adversarial training to ADM | ADM · CIFAR-10 32x32, DDIM, 5 NFEs | FID: 29.30 -> 26.38 | Helped |
| Apply adversarial training to ADM | ADM · CIFAR-10 32x32, IDDPM, 5 NFEs | FID: 36.91 -> 37.15 | Hurt |
| Apply adversarial training to ADM | ADM · ImageNet 64x64, DPM-Solver, 5 NFEs | FID: 27.82 -> 17.36 | Helped |
| Apply adversarial training to ADM | ADM · ImageNet 64x64, DPM-Solver, 50 NFEs | FID: 4.15 -> 4.34 | Hurt |
| Apply adversarial training to ADM | ADM · ImageNet 64x64, DDIM, 5 NFEs | FID: 76.51 -> 43.04 | Helped |
| Apply adversarial training to LCM | LCM · MS-COCO 2014 validation, 512x512, 1 step | FID: 25.43 -> 23.34 | Helped |
| Apply adversarial training to LCM | LCM · MS-COCO 2014 validation, 512x512, 8 steps | FID: 12.62 -> 10.68 | Helped |
| Apply adversarial training to LCM | LCM · MS-COCO 2014 validation, 512x512, 1 step | CLIP score: 29.25 -> 29.63 | Helped |
| Apply adversarial training to LCM | LCM · MS-COCO 2014 validation, 512x512, 8 steps | CLIP score: 30.47 -> 30.53 | Helped |
| Use efficient adversarial training instead of PGD-based standard adversarial training | Diffusion models · Training speed | speed-up: 2.6× speed-up | Helped |
Surprising findings
- Plain IP-style Gaussian input perturbation (ADM-IP) can hurt FID at few sampling steps (e.g., 43.83 vs 23.95 for DPM-Solver at 5 NFEs on CIFAR-10).
Limitations
- Experiments fine-tune pretrained models rather than training from scratch for DPM; from-scratch results are only in the appendix.
- Theoretical results rely on regularity conditions and approximations (e.g., Gaussian conditional approximations, o(1) terms).
Experimental setup
Datasets: CIFAR-10 ImageNet 64x64 LAION-Aesthetics-6.5+ MS-COCO 2014
Benchmarks: CIFAR-10 32x32 ImageNet 64x64 MS-COCO 2014 512x512
Metrics: FID CLIP score
Baselines: ADM (original) ADM (finetune) ADM-IP LCM IDDPM DDIM DPM-Solver ES
Builds on: DDPM ADM Consistency Models Free-AT Latent Consistency Models
Closest papers by meaning
- On Inference Stability for Diffusion Models (AAAI 2024 · similarity 0.90)
- Bridging Data Gaps in Diffusion Models with Adversarial Noise-Based Transfer Learning (ICML 2024 · similarity 0.90)
- ACT-Diffusion: Efficient Adversarial Consistency Training for One-step Diffusion Models (CVPR 2024 · similarity 0.89)
- Robust Classification via a Single Diffusion Model (ICML 2024 · similarity 0.88)
- Consistency-GAN: Training GANs with Consistency Model (AAAI 2024 · similarity 0.88)
- Evaluating the design space of diffusion-based generative models (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.