Research map / Image generation / GANs
Bridging Data Gaps in Diffusion Models with Adversarial Noise-Based Transfer Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DPMs-ANT, a diffusion transfer learning method for few-shot image generation that combines classifier-guided similarity training with adversarial (worst-case) noise selection and trains only a small adaptor on a pretrained DDPM/LDM.
Problem
Transferring pretrained diffusion models to target domains with very little data (10 images) overfits or underfits, and GAN-based few-shot transfer methods do not map cleanly onto the iterative noisy-image denoising process of DPMs.
Approach
Guides the fine-tuning of a pretrained DPM with the gradient of a fixed source-vs-target binary classifier on noised images (similarity-guided training), and selects per-sample worst-case Gaussian noise via multi-step gradient ascent (adversarial noise selection), optimizing only lightweight adaptor layers.
Key results
- DDPM-ANT reaches FID 20.06 on FFHQ to 10-shot Sunglasses (vs 34.75 for DDPM-PA) and 46.70 on Babies (vs 48.92), with FID computed on larger target sets.
- Intra-LPIPS on LSUN Church to Landscape drawings improves from 0.706 (DDPM-PA) to 0.723 (DDPM-ANT) and LDM-ANT reaches 0.738, while fine-tuning only ~1.3-1.6% of parameters.
- Ablation on Sunglasses: FID drops from 41.88 (adaptor only) to 26.41 (similarity guidance) to 20.06 (full DPMs-ANT); training takes about 3 GPU hours versus 4.2 for the 5,000-iteration baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DDPM-ANT instead of DDPM-PA | DDPM transfer learning · FFHQ → Babies (10-shot) | FID: 48.92 -> 46.70 | Helped |
| Use DDPM-ANT instead of DDPM-PA | DDPM transfer learning · FFHQ → Sunglasses (10-shot) | FID: 34.75 -> 20.06 | Helped |
| Use DDPM-ANT instead of DDPM-PA | 10-shot diffusion transfer · FFHQ → Sunglasses | Intra-LPIPS: 0.604±0.014 -> 0.613±0.023 | Helped |
| Use DDPM-ANT instead of DDPM-PA | 10-shot diffusion transfer · FFHQ → Raphael’s paintings | Intra-LPIPS: 0.581±0.041 -> 0.621±0.068 | Helped |
| Use DDPM-ANT instead of DDPM-PA | 10-shot diffusion transfer · LSUN Church → Haunted houses | Intra-LPIPS: 0.628±0.029 -> 0.648±0.010 | Helped |
| Use DDPM-ANT instead of DDPM-PA | 10-shot diffusion transfer · LSUN Church → Landscape drawings | Intra-LPIPS: 0.706±0.030 -> 0.723±0.020 | Helped |
| Use DDPM-ANT instead of DDPM-PA | 10-shot diffusion transfer · FFHQ → Babies | Intra-LPIPS: 0.599±0.024 -> 0.592±0.016 | Hurt |
| Add similarity-guided training (ablation) | DDPM-ANT with adaptor · FFHQ → Sunglasses (10-shot) | FID: 41.88 -> 26.41 | Helped |
| Add adversarial noise selection (ablation) | DPMs-ANT with similarity-guided training · FFHQ → Sunglasses (10-shot) | FID: 26.41 -> 20.06 | Helped |
| Fine-tune only the adaptor layer instead of the full model (ablation) | DDPM transfer learning · FFHQ → Sunglasses (10-shot) | FID: 38.65 -> 41.88 | Hurt |
| Train the classifier on 100 rather than 10 images (ablation) | DPMs-ANT · FFHQ → Sunglasses (10-shot) | Intra-LPIPS: 0.613 ± 0.023 -> 0.637 ± 0.013 | Helped |
| Train the classifier on 100 rather than 10 images (ablation) | DPMs-ANT · FFHQ → Sunglasses (10-shot) | FID: 20.06 -> 22.84 | Hurt |
Surprising findings
- A classifier trained on only 100 target images gives worse FID (22.84) than the 10-shot classifier (20.06), suggesting more classifier data does not help.
Limitations
- Generated images carry target-domain specifics, risking inconsistency and privacy leakage of sensitive target information (e.g., reflections in sunglasses).
- FID is unstable on 10-shot datasets, so FID was computed on larger target sets (2.5k-2.7k images) instead.
Experimental setup
Datasets: FFHQ LSUN Church ImageNet 10-shot Babies 10-shot Sunglasses 10-shot Raphael Peale 10-shot Sketches 10-shot Modigliani face paintings 10-shot Haunted Houses 10-shot Landscape drawings
Benchmarks: 10-shot FFHQ to Babies/Sunglasses/Raphael's paintings 10-shot LSUN Church to Haunted houses/Landscape drawings
Metrics: Intra-LPIPS FID LPIPS
Baselines: TGAN TGAN+ADA EWC CDC DCL DDPM-PA
Builds on: DDPM DDIM classifier guidance DDPM-PA adapter layers
Closest papers by meaning
- Improved Diffusion-based Generative Model with Better Adversarial Robustness (ICLR 2025 · similarity 0.90)
- Learning to Discretize Denoising Diffusion ODEs (ICLR 2025 · similarity 0.89)
- Revisiting Diffusion Models: From Generative Pre-training to One-Step Generation (ICML 2025 · similarity 0.89)
- You Only Sample Once: Taming One-Step Text-to-Image Synthesis by Self-Cooperative Diffusion GANs (ICLR 2025 · similarity 0.89)
- UDPM: Upsampling Diffusion Probabilistic Models (NeurIPS 2024 · similarity 0.89)
- Structure-Guided Adversarial Training of Diffusion Models (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.