Research map / Image generation / Fast diffusion sampling
Reducing Spatial Fitting Error in Distillation of Denoising Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SFERD, a diffusion distillation framework that reduces spatial fitting error in teacher (attention-guided denoising targets) and student (semantic gradient predictor with learned latent encoder), reaching 5.31 FID on CIFAR-10 and 9.39 on ImageNet 64x64 in one step.
Problem
Knowledge distillation of diffusion models for few-step sampling suffers degraded generation quality, which the authors attribute to spatial fitting error in both teacher and student training.
Approach
Use teacher self-attention maps to identify high-variance "Risky Regions" and apply Gaussian-blur-based attention guidance to correct the teacher's denoised target, and add a learned semantic latent encoder plus gradient predictor so the student gets extra reconstruction information during distillation.
Key results
- SFERD-CD achieves FID 5.31 on CIFAR-10 and 9.39 on ImageNet 64x64 with 1 NFE, versus CD's 8.21 and 13.87.
- SFERD-CD with 4 steps reaches FID 2.68 on CIFAR-10 and 4.41 on ImageNet 64x64, improving over CD (3.39 and 5.81).
- In a human study on Bedroom 256x256, 52.8% of participants preferred SFERD-CD samples over CD at 4 steps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add SFERD attention guidance and semantic gradient predictor to PD | PD · CIFAR-10, 1–4 steps | FID: 14.85 -> 7.54; 7.64 -> 6.37; 4.28 -> 3.44 | Helped |
| add SFERD attention guidance and semantic gradient predictor to CD | CD · CIFAR-10, 1–4 steps | FID: 8.21 -> 5.31; 6.26 -> 4.19; 3.39 -> 2.68 | Helped |
| add SFERD attention guidance and semantic gradient predictor to PD | PD · ImageNet 64×64, 1–4 steps | FID: 19.23 -> 14.85; 9.71 -> 7.53; 7.22 -> 5.93 | Helped |
| add SFERD attention guidance and semantic gradient predictor to CD | CD · ImageNet 64×64, 1–4 steps | FID: 13.87 -> 9.39; 8.24 -> 6.08; 5.81 -> 4.41 | Helped |
| add SFERD improvements to EDM | EDM · CIFAR-10, 35 NFE | FID: 2.41 -> 2.12 | Helped |
| add SFERD improvements to EDM | EDM · ImageNet 64×64, 79 NFE | FID: 2.99 -> 2.43 | Helped |
| add SFERD attention guidance to CD | CD · Bedroom 256×256, 1–4 steps | FID: 16.09 -> 9.38; 6.37 -> 5.62; 5.59 -> 4.92 | Helped |
| use 2 distillation stages instead of 1 (ablation) | SFERD-PD · CIFAR-10, final sampling step 1 | FID: 16.79 -> 7.54 | Helped |
| use 4 distillation stages instead of 2 (ablation) | SFERD-PD · CIFAR-10, final sampling step 1 | FID: 7.54 -> 9.31 | Hurt |
| use 2 distillation stages instead of 1 (ablation) | SFERD-PD · CIFAR-10, final sampling step 2 | FID: 14.22 -> 6.37 | Helped |
| use 4 distillation stages instead of 2 (ablation) | SFERD-PD · CIFAR-10, final sampling step 2 | FID: 6.37 -> 8.28 | Hurt |
| set attention threshold to 1.0 rather than a higher or lower value (ablation) | SFERD-PD · ImageNet 64×64, 4 steps | FID | Helped |
Surprising findings
- Fitting error of the teacher is dominated by prediction variance across sampling, and the Pearson correlation with self-attention maps strengthens as t goes from 1 to 0.
- Gaussian blurring of the denoised prediction brings it closer to the final sample in most timesteps, which the authors use to justify their design.
Limitations
- Gains on Bedroom 256 are limited to a single distillation implementation (SFERD-CD) due to compute constraints.
- Human evaluation shows only a small preference margin (52.8%).
- Attention guidance requires hyperparameter tuning (threshold, blur sigma, guidance strength).
Experimental setup
Datasets: CIFAR-10 ImageNet 64x64 LSUN Bedroom 256x256
Benchmarks: CIFAR-10 ImageNet 64x64 LSUN Bedroom 256x256
Metrics: FID IS l2 distance fitting error
Baselines: PD CD DFNO ViTGAN DiffAugment-BigGAN TransGAN EDM DDIM StyleGAN2 PG-SWGAN
Builds on: Progressive Distillation Consistency Distillation DDIM EDM classifier guidance
Closest papers by meaning
- Simple Distillation for One-Step Diffusion Models (NeurIPS 2025 · similarity 0.90)
- Noise Conditional Variational Score Distillation (ICML 2025 · similarity 0.90)
- Simple and Fast Distillation of Diffusion Models (NeurIPS 2024 · similarity 0.90)
- Denoising Diffusion Bridge Models (ICLR 2024 · similarity 0.90)
- Diffusion Model is Effectively Its Own Teacher (CVPR 2025 · similarity 0.89)
- EVODiff: Entropy-aware Variance Optimized Diffusion Inference (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.