Research map / Image generation / Fast diffusion sampling

Diffusion Tuning: Transferring Diffusion Models via Chain of Forgetting

NeurIPS 2024 · Image generation › Fast diffusion sampling · approach: Diffusion models as perception tools · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Diff-Tuning, a fine-tuning method for diffusion models that exploits a "chain of forgetting" in the reverse process by pairing knowledge retention (on pre-sampled data from the pre-trained model) with time-weighted knowledge reconsolidation on the target data.

Problem

Standard fine-tuning of pre-trained diffusion models forgets general denoising knowledge and overfits downstream data, while PEFT methods for diffusion models are not tailored to diffusion's multi-step structure.

Approach

Observes that a pre-trained diffusion model's zero-shot denoising transferability decreases monotonically along the reverse process (chain of forgetting), and uses time-dependent loss weights so that the fine-tuned model retains pre-trained knowledge at small t (retention on pre-sampled data) and adapts to the new domain at large t (reconsolidation on target data).

Key results

• Diff-Tuning achieves average FID 10.63 across 8 downstream datasets vs 14.36 for full fine-tuning (26.0% relative improvement) and 12.81 for DiffFit.
• On ControlNet with Stable Diffusion, Diff-Tuning reduces average sudden convergence steps from 9.0k to 6.6k (about 24% faster) across five conditions.
• Applying Diff-Tuning to DiffFit (Diff-Tuning*) improves its converged performance, and DiffFit's EWC values are reported as 2.42x larger than Diff-Tuning's per the paper's analysis.

Measured effects reported in the paper

Surprising findings

• Using the entire 1.2M ImageNet source dataset for knowledge retention gives suboptimal results compared with a smaller pre-sampled set from the pre-trained model.
• Replacing the last denoising steps with the original pre-trained model can achieve competitive or slightly better quality than the fine-tuned model.

Limitations

• Requires constructing a pre-sampled augmented dataset from the pre-trained model, adding computational cost.
• Introduces extra hyperparameters (retention and reconsolidation weighting functions) whose design is heuristic.
• Similarity-based sudden convergence metric is imperfect and needs human assessment; similarity scores may not reflect quality.

Experimental setup

Datasets: ImageNet Food101 SUN397 DF20-Mini Caltech101 CUB-200-2011 ArtBench-10 Oxford Flowers Stanford Cars COCO ADE20k

Benchmarks: Food101 SUN397 DF20-Mini Caltech101 CUB-200-2011 ArtBench-10 Oxford Flowers Stanford Cars COCO ADE20k

Metrics: FID-10K average FID relative improvement sudden convergence steps SSIM MSE pixel accuracy EWC value

Baselines: Full Fine-tuning AdaptFormer BitFit VPT-Deep LoRA DiffFit ControlNet

Builds on: DiT DiffFit ControlNet Elastic Weight Consolidation DDIM

Closest papers by meaning

• Domain Guidance: A Simple Transfer Approach for a Pre-trained Diffusion Model (ICLR 2025 · similarity 0.89)
• Stochastic Control for Fine-tuning Diffusion Models: Optimality, Regularity, and Convergence (ICML 2025 · similarity 0.89)
• Transfer Learning for Diffusion Models (NeurIPS 2024 · similarity 0.89)
• Efficient Fine-Tuning and Concept Suppression for Pruned Diffusion Models (CVPR 2025 · similarity 0.89)
• Distilling ODE Solvers of Diffusion Models into Smaller Steps (CVPR 2024 · similarity 0.88)
• Diffusion Model Patching via Mixture-of-Prompts (AAAI 2025 · similarity 0.88)