Research map / Image generation / Text-to-image diffusion
Image is All You Need to Empower Large-scale Diffusion Models for In-Domain Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts Stable Diffusion v1.5 to image-only domains (faces, animals, porcelain) via guidance-decoupled prior preservation and a null-text domain UNet, enabling unconditional, text, spatial, editing and 3D generation.
Problem
Fine-tuning large diffusion models on label-free domain image data improves fidelity but causes catastrophic forgetting of text/control guidance and unconditional guidance, hurting controllability.
Approach
Decouple the conditional guidance in classifier-free guidance into a domain guidance (learned by a separate text-free null-text UNet copy) and open-world control guidance, while keeping unconditional and control guidance predicted by the frozen pre-trained diffusion prior, combined in a multi-guidance sampling formula.
Key results
- Unconditional generation FID on face domain drops to 6.57 (vs 12.37 for vanilla fine-tuning) with 87.5% human preference.
- Text-guided alignment score of 0.89 vs 0.97 for Textual Inversion (which has higher alignment but much lower quality).
- Animal domain FID 18.82 vs 23.92 for vanilla fine-tuning, with 80.2% human preference.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use guidance-decoupled prior preservation with domain guidance learned by a separate null-text model | Stable Diffusion v1.5 · Face | FID: 12.37 -> 6.57 | Helped |
| Use guidance-decoupled prior preservation with domain guidance learned by a separate null-text model | Stable Diffusion v1.5 · Animal | FID: 23.92 -> 18.82 | Helped |
| Use guidance-decoupled prior preservation with domain guidance learned by a separate null-text model | Stable Diffusion v1.5 · Porcelain | FID: 62.76 -> 56.46 | Helped |
| Use guidance-decoupled prior preservation | Stable Diffusion v1.5 · Text-Guided | Align: 0.97 -> 0.89 | Hurt |
| Rectify unconditional guidance using the pretrained diffusion prior | In-domain generation and open-sourced customized diffusion models · Face domain generation | Image quality | Helped |
| Fine-tune the pretrained diffusion model on domain data | Stable Diffusion v1.5 · Face domain text-guided generation | Controllability | Hurt |
Surprising findings
- Textual Inversion achieves higher text alignment than the proposed method, at a cost to image quality.
- Unconditional guidance rectification using the original pre-trained model also improves open-source customized models such as Counterfeit and Dreamshaper.
Limitations
- Text-alignment is lower than Textual Inversion on text-guided generation.
- Multi-guidance assumes independent conditions, which can cause confusion between conditions; mitigated by detailed prompts.
Experimental setup
Datasets: FFHQ AFHQv2 porcelain dataset (collected from internet) CelebA-HQ
Benchmarks: FFHQ AFHQv2 porcelain dataset CelebA-HQ
Metrics: FID human preference win rate attribute alignment canny discrepancy
Baselines: Textual Inversion Custom Diffusion OFT Vanilla fine-tuning LoRA DreamFusion
Builds on: Stable Diffusion classifier-free guidance Textual Inversion ControlNet DreamFusion
Closest papers by meaning
- Domain Guidance: A Simple Transfer Approach for a Pre-trained Diffusion Model (ICLR 2025 · similarity 0.93)
- Elucidating the design space of classifier-guided diffusion generation (ICLR 2024 · similarity 0.89)
- When Model Knowledge meets Diffusion Model: Diffusion-assisted Data-free Image Synthesis with Alignment of Domain and Class (ICML 2025 · similarity 0.89)
- Plug-and-Play Diffusion Distillation (CVPR 2024 · similarity 0.89)
- Learning Diffusion Models with Flexible Representation Guidance (NeurIPS 2025 · similarity 0.88)
- DKDM: Data-Free Knowledge Distillation for Diffusion Models with Any Architecture (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.