Research map / Image generation / Fast diffusion sampling
The Surprising Effectiveness of Skip-Tuning in Diffusion Sampling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Skip-Tuning, a training-free method that scales down UNet encoder-to-decoder skip connections in pretrained diffusion models, improving few-step and many-step ODE sampling FID (e.g., EDM on ImageNet 64 from 3.64 to 1.88 at 19 NFE).
Problem
Skip connections in diffusion UNets may limit the complexity of the transformation from Gaussian noise to data, hurting few-step diffusion sampling quality, and ODE samplers appear to saturate at a FID limit regardless of step count.
Approach
Multiply the down-sampling skip vectors by a coefficient rho<1 (linearly interpolated across layers) before concatenation in a pretrained UNet, a post-training change requiring no retraining, which reduces effective skip strength and improves sampling; the gains are explained via lower feature-space score-matching loss at intermediate noise levels.
Key results
- On ImageNet 64, EDM with Skip-Tuning (rho 0.82 to 1.0) reaches FID 1.75 at 19 NFE with UniPC, versus 2.60 for the baseline.
- At 39 NFE, EDM with Skip-Tuning reaches FID 1.57, slightly beating EDM-2-S (1.58) and the baseline EDM at 2.21.
- Skip-Tuning on distilled one-step models improves CD EDM FID from 6.85 to 5.56 and DI EDM from 4.16 to 3.98.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| scale skip connections (ρ: 0.78 to 1.0) | EDM · ImageNet 64, Heun, 10 steps | FID: 3.64 -> 1.88 | Helped |
| scale skip connections (ρ: 0.82 to 1.0) | EDM · ImageNet 64, UniPC, 19 steps | FID: 2.60 -> 1.75 | Helped |
| scale skip connections (ρ: 0.83 to 1.0) | EDM · ImageNet 64, UniPC, 39 steps | FID: 2.21 -> 1.57 | Helped |
| scale skip connections | CD EDM · ImageNet 64, one-step | FID: 6.85 -> 5.56 | Helped |
| scale skip connections | DI EDM · ImageNet 64, one-step | FID: 4.16 -> 3.98 | Helped |
| scale skip connections | EDM · AFHQv2 64×64, UniPC | FID: 4.47 -> 3.85; 2.13 -> 2.03; 2.05 -> 1.96 | Helped |
| scale skip connections | LDM and UViT · ImageNet 256×256 | FID: 12.97 -> 11.29; 4.91 -> 4.67; 4.25 -> 4.13; 2.32 -> 2.21 | Helped |
| fine-tune ρ with ResNet-101 feature score-matching loss without a Sigmoid constraint (ablation) | EDM · ImageNet 64, 5-step sampling | FID: 35.12 -> 215.09 | Hurt |
| constrain fine-tuned ρ with Sigmoid (ablation) | EDM · ImageNet 64, 5-step sampling | FID: 215.09 -> 18.92 | Helped |
| constrain fine-tuned ρ with Sigmoid instead of unconstrained ρ fine-tuning (ablation) | EDM · ImageNet 64, 10-step sampling | FID: 112.15 -> 2.77 | Helped |
| fine-tune all parameters with hybrid pixel and Inception-V3 feature loss (ablation) | EDM · ImageNet 64, 19 NFE | FID: 2.60141 -> 2.51128 -> 3.81702 -> 6.02844 | Mixed |
| scale only the UNet block orig variable instead of applying skip tuning to the concatenation (ablation) | EDM · ImageNet 64, 39 NFE | FID: 1.57 -> 1.58 | No clear effect |
Surprising findings
- Skip-Tuning increases pixel-space score-matching loss while decreasing feature-space score-matching loss (ResNet-101, Inception-V3, CLIP), especially at intermediate noise levels.
- Directly fine-tuning the skip coefficients or the full network with a feature-space loss degrades sampling quality, while the training-free tuning improves it.
Limitations
- Optimal rho depends on sampler, NFE, and model, and must be found by search on the sweet-point range.
- Recall decreases slightly with Skip-Tuning (0.639 to 0.625).
- Not studied on transformer-based diffusion models such as DiT that lack long skip connections.
Experimental setup
Datasets: ImageNet 64 ImageNet 256 AFHQv2 64
Benchmarks: ImageNet 64 ImageNet 256 AFHQv2 64
Metrics: FID Inception Score Precision Recall MMD IMMD
Baselines: EDM Heun sampler EDM UniPC sampler EDM-2-S FreeU CD EDM DI EDM LDM UViT
Builds on: EDM UniPC FreeU Diff-Instruct Consistency Models
Closest papers by meaning
- SCEdit: Efficient and Controllable Image Diffusion Generation via Skip Connection Editing (CVPR 2024 · similarity 0.90)
- SkipDiff: Adaptive Skip Diffusion Model for High-Fidelity Perceptual Image Super-resolution (AAAI 2024 · similarity 0.89)
- Analyzing and Improving the Training Dynamics of Diffusion Models (CVPR 2024 · similarity 0.88)
- A Simple Early Exiting Framework for Accelerated Sampling in Diffusion Models (ICML 2024 · similarity 0.88)
- Spatiotemporal Skip Guidance for Enhanced Video Diffusion Sampling (CVPR 2025 · similarity 0.88)
- SaRA: High-Efficient Diffusion Model Fine-tuning with Progressive Sparse Low-Rank Adaptation (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.