Research map / Image generation / Fast diffusion sampling
Domain Guidance: A Simple Transfer Approach for a Pre-trained Diffusion Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Domain Guidance (DoG), a transfer method for pre-trained diffusion models that extrapolates a fine-tuned model away from the frozen pre-trained model, and shows FID and FD_DINOv2 gains over CFG-style fine-tuning on seven transfer benchmarks.
Problem
Fine-tuning large pre-trained diffusion models to target domains suffers from catastrophic forgetting and poor fitness of the unconditional guiding model in standard classifier-free guidance, producing out-of-distribution samples.
Approach
Treat transfer as conditioning on the target domain relative to the pre-trained distribution: fine-tune only the conditional branch and guide sampling with the frozen pre-trained model as the unconditional reference, using a DoG factor analogous to the CFG weight. Existing CFG-fine-tuned models can use it without retraining.
Key results
- Domain guidance reduces average FID from 13.69 (CFG) to 11.55, a 19.59% relative improvement over CFG on seven transfer benchmarks.
- Average FD_DINOv2 falls from 363.58 (CFG) to 292.62, a 23.4% relative improvement.
- On SDXL with off-the-shelf LoRAs, CLIP score on the Chalkboard style rises from 27.23 (CFG) to 35.24 (DoG).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace CFG with Domain Guidance | DiT-XL/2-256x256 transfer · Seven downstream datasets, average | FID: 13.69 -> 11.55; relative promotion 19.59% | Helped |
| Replace CFG with Domain Guidance | DiT-XL/2-256x256 transfer · Seven downstream datasets, average | FD_DINOv2: 363.58 -> 292.62; 23.4% | Helped |
| Replace CFG with Domain Guidance | SDXL off-the-shelf LoRA transfer · Chalkboard Drawing Style | CLIP Score: 27.23 -> 35.24 | Helped |
| Replace CFG with Domain Guidance | SDXL off-the-shelf LoRA transfer · Yarn Art Style | CLIP Score: 34.89 -> 35.03 | Helped |
| Use a separately fine-tuned unconditional guiding model, increasing its training steps (ablation) | Fine-tuned conditional model on CUB · CUB-200-2011 | FID | Hurt |
| Remove unconditional training and use 21,600 conditional training steps (ablation) | Domain transfer training · Caltech101, DF20-Mini, and ArtBench-10 | Generation quality | Helped |
| Vary the DoG guidance weight (ablation) | Domain Guidance · Various downstream datasets | FID: 1.4 to 2 | Mixed |
| Vary the sampling-step count (ablation) | Domain Guidance · Downstream transfer benchmarks | Generation performance | Helped |
| Use Domain Guidance instead of CFG | DiT-XL/2-256x256 transfer · Seven downstream datasets | Precision: 0.559 -> 0.646 (average) | Helped |
| Use Domain Guidance instead of CFG | DiT-XL/2-256x256 transfer · Seven downstream datasets | Recall: 0.604 -> 0.590 (average) | Mixed |
Surprising findings
- Fine-tuning a separate unconditional guiding model for longer degrades guidance performance as training steps increase, attributed to catastrophic forgetting and overfitting.
- Fine-tuned DiT-XL/2 on Stanford Cars has FID 6.57 conditionally but 22.8 unconditionally, showing the unconditional branch fits poorly.
Limitations
- Guidance weight must be tuned, with best results typically in a narrow range of 1.4 to 2.
- Theoretical analysis is limited to a bound on marginal estimation error and a reformulation as classifier guidance; the 2D toy example is small-scale.
- Evaluation is limited to class-conditional DiT-XL/2 on fine-grained datasets plus a small SDXL LoRA study with CLIP score only.
Experimental setup
Datasets: Food101 SUN397 DF20-Mini Caltech101 CUB-200-2011 ArtBench-10 Stanford Cars ImageNet
Benchmarks: Food101 SUN397 DF20-Mini Caltech101 CUB-200-2011 ArtBench-10 Stanford Cars
Metrics: FID FD_DINOv2 precision recall CLIP score
Baselines: Fine-tuning without guidance Classifier-free guidance fine-tuning Autoguidance DiffFit
Builds on: classifier-free guidance classifier guidance DiT Stable Diffusion XL
Closest papers by meaning
- Transfer Learning for Diffusion Models (NeurIPS 2024 · similarity 0.93)
- Image is All You Need to Empower Large-scale Diffusion Models for In-Domain Generation (CVPR 2025 · similarity 0.93)
- Token Perturbation Guidance for Diffusion Models (NeurIPS 2025 · similarity 0.92)
- TFG: Unified Training-Free Guidance for Diffusion Models (NeurIPS 2024 · similarity 0.91)
- No Training, No Problem: Rethinking Classifier-Free Guidance for Diffusion Models (ICLR 2025 · similarity 0.91)
- Elucidating the design space of classifier-guided diffusion generation (ICLR 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.