Research map / Image generation / GANs
Structure-Guided Adversarial Training of Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SADM, which adds a batch-level manifold structure loss and an adversarially trained structure discriminator to diffusion model training, improving FID on image generation and cross-domain fine-tuning.
Problem
Diffusion models are trained with instance-level denoising losses that ignore pair-wise structural relationships among samples within a mini-batch, which may limit how well the full data distribution is captured.
Approach
Align the affinity (pair-wise similarity) matrix of predicted clean samples with that of ground-truth samples in each batch, and train this structure term adversarially against a comparison-based structure discriminator that uses a pre-trained Inception encoder in a minimax game with the denoiser.
Key results
- Achieves FID 1.58 on ImageNet 256x256 and 2.11 on ImageNet 512x512 class-conditional generation with DiT-SADM, versus 2.27 and 3.04 for DiT-XL/2-G.
- On CIFAR-10 with EDM backbone in data space, SADM reaches FID 1.54 unconditional and 1.47 conditional, improving over EDM-G++ (1.77 and 1.64).
- In cross-domain fine-tuning from ImageNet, SADM (full) averages FID 14.15 across 8 target datasets versus 16.59 for full fine-tuning with DDPM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use structure-guided adversarial training instead of standard DiT training | DiT · ImageNet 256×256 | FID: 2.27 -> 1.58 | Helped |
| Use structure-guided adversarial training instead of standard DiT training | DiT · ImageNet 512×512 | FID: 3.04 -> 2.11 | Helped |
| Use SADM with data-space diffusion instead of EDM-G++ | EDM · CIFAR-10 unconditional | FID: 1.77 -> 1.54 | Helped |
| Use structure-guided training with a DiT for fine-tuning | DiT pretrained on ImageNet · Cross-domain fine-tuning average | FID: 15.39 -> 14.27 | Helped |
| Add structural guidance (ablation) | DDPM · ImageNet | FID: 4.59 -> 3.57 | Helped |
| Add a structure discriminator to structural guidance (ablation) | DDPM · ImageNet | FID: 3.57 -> 3.14 | Helped |
| Add structural guidance (ablation) | DDPM · CIFAR-10 | FID: 3.17 -> 2.64 | Helped |
| Add a structure discriminator to structural guidance (ablation) | DDPM · CIFAR-10 | FID: 2.64 -> 2.33 | Helped |
| Add structural guidance (ablation) | DDPM · CelebA | FID: 2.32 -> 1.82 | Helped |
| Add a structure discriminator to structural guidance (ablation) | DDPM · CelebA | FID: 1.82 -> 1.64 | Helped |
Limitations
- Structure discriminator relies on an off-the-shelf pre-trained Inception-V3 encoder, which may not give meaningful embeddings for noisy inputs.
- Evaluated only on image generation and fine-tuning; text-to-image and video are left to future work.
- Adversarial tuning adds training rounds and hyperparameters (eta_t, gamma) that must be balanced.
Experimental setup
Datasets: ImageNet CIFAR-10 CelebA FFHQ Food101 SUN397 DF-20M mini Caltech101 CUB-200-2011 ArtBench-10 Oxford Flowers Stanford Cars
Benchmarks: ImageNet 256x256 ImageNet 512x512 CIFAR-10 CelebA 64x64 FFHQ 64x64 Food101 SUN397 DF-20M mini Caltech101 CUB-200-2011 ArtBench-10 Oxford Flowers
Metrics: FID Inception Score Precision Recall NLL
Baselines: ADM-G StyleGAN-XL BigGAN-deep LDM-4-G DiT-XL/2-G MDT-G DiT-G++ EDM-G++ LSGM-G++ DiffFit
Builds on: DiT EDM LSGM ADM classifier guidance DDPM
Closest papers by meaning
- Revisiting Diffusion Models: From Generative Pre-training to One-Step Generation (ICML 2025 · similarity 0.90)
- Efficient Image-to-Image Diffusion Classifier for Adversarial Robustness (AAAI 2025 · similarity 0.90)
- Adapting to Unknown Low-Dimensional Structures in Score-Based Diffusion Models (NeurIPS 2024 · similarity 0.89)
- Generalization in diffusion models arises from geometry-adaptive harmonic representations (ICLR 2024 · similarity 0.89)
- Unlocking Dataset Distillation with Diffusion Models (NeurIPS 2025 · similarity 0.89)
- Manifold Preserving Guided Diffusion (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.