Research map / Image generation / Editing and style transfer
Smooth Diffusion: Crafting Smooth Latent Spaces in Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Smooth Diffusion, a Stable Diffusion variant trained with a Step-wise Variation Regularization (via Smooth-LoRA) so that latent perturbations yield proportional image changes, measured by a new ISTD metric.
Problem
Diffusion model latent spaces are non-smooth: small latent perturbations cause large, irregular visual fluctuations, which hurts interpolation, inversion, and editing.
Approach
Adds a step-wise regularizer that bounds the Jacobian of the denoised output w.r.t. the initial noise at every diffusion training step, so output variation is proportional to input variation; also introduces the ISTD metric to quantify latent smoothness.
Key results
- ISTD drops from 38.63 (Stable Diffusion) to 16.54 (Smooth Diffusion), while FID improves from 12.70 to 12.10 and CLIP Score rises from 31.46 to 31.54.
- With DDIM inversion, PSNR improves from 13.97 (Stable Diffusion) to 16.17 (Smooth Diffusion); with NTI, PSNR is 25.63 vs 25.74.
- Ablation: regularization ratio 10 gives ISTD 11.51 but FID rises to 17.44; full fine-tuning gives ISTD 11.52 but FID 27.27.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Step-wise Variation Regularization | Stable Diffusion v1.5 · MS-COCO val latent interpolation | ISTD: 38.63 -> 16.54 | Helped |
| Add Step-wise Variation Regularization | Stable Diffusion v1.5 · MS-COCO val text-to-image generation | FID: 12.70 -> 12.10 | Helped |
| Add Step-wise Variation Regularization | Stable Diffusion v1.5 · MS-COCO val image reconstruction with DDIM inversion | MSE: 0.1756 -> 0.1086 | Helped |
| Add Step-wise Variation Regularization | Stable Diffusion v1.5 · MS-COCO val image reconstruction with Null-text inversion | MSE: 0.0156 -> 0.0153 | Helped |
| Set regularization ratio λ to 0.1 (ablation) | Smooth Diffusion · MS-COCO val | ISTD: 16.54 -> 24.23 | Hurt |
| Set regularization ratio λ to 0.1 (ablation) | Smooth Diffusion · MS-COCO val | CLIP Score: 31.49 -> 31.56 | Helped |
| Set regularization ratio λ to 10 (ablation) | Smooth Diffusion · MS-COCO val | ISTD: 16.54 -> 11.51 | Helped |
| Set regularization ratio λ to 10 (ablation) | Smooth Diffusion · MS-COCO val | FID: 12.11 -> 17.44 | Hurt |
| Use LoRA rank 4 (ablation) | Smooth Diffusion · MS-COCO val | ISTD: 16.54 -> 16.76 | Hurt |
| Use LoRA rank 16 (ablation) | Smooth Diffusion · MS-COCO val | FID: 12.11 -> 11.49 | Helped |
| Fully fine-tune instead of using LoRA (ablation) | Smooth Diffusion · MS-COCO val | FID: 12.11 -> 27.27 | Hurt |
| Fully fine-tune instead of using LoRA (ablation) | Smooth Diffusion · MS-COCO val | ISTD: 16.54 -> 11.52 | Helped |
Surprising findings
- Full fine-tuning yields lower ISTD but substantially degrades image quality (FID 27.27, CLIP 28.86), so LoRA is needed for a good trade-off.
Limitations
- Stronger regularization improves smoothness but increases FID, so a trade-off is required.
- Full fine-tuning collapses in quality under the default setting.
- Editing results rely on a simple NTI-based pipeline and are evaluated qualitatively.
Experimental setup
Datasets: LAION Aesthetics 6.5+ MS-COCO validation set
Benchmarks: MS-COCO validation
Metrics: ISTD FID CLIP Score MSE LPIPS SSIM PSNR
Baselines: Stable Diffusion ANID VAE interpolation SDEdit Prompt-to-Prompt Plug-and-Play Pix2Pix-Zero Cycle Diffusion DragDiffusion
Builds on: Stable Diffusion LoRA Null-text inversion DDIM inversion StyleGAN2 path length regularization
Closest papers by meaning
- Simpler Diffusion: 1.5 FID on ImageNet512 with Pixel-space Diffusion (CVPR 2025 · similarity 0.91)
- Tracing the Roots: Leveraging Temporal Dynamics in Diffusion Trajectories for Origin Attribution (NeurIPS 2025 · similarity 0.90)
- Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference (NeurIPS 2024 · similarity 0.89)
- Multistep Distillation of Diffusion Models via Moment Matching (NeurIPS 2024 · similarity 0.89)
- Alias-Free Latent Diffusion Models: Improving Fractional Shift Equivariance of Diffusion Latent Space (CVPR 2025 · similarity 0.89)
- DisCo-Diff: Enhancing Continuous Diffusion Models with Discrete Latents (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.