Research map / Image generation / Text-to-image diffusion
Clockwork Diffusion: Efficient Generation With Model-Step Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Clockwork Diffusion speeds up text-to-image diffusion UNets by replacing low-resolution UNet computation with a lightweight adaptor that reuses representations from previous sampling steps, alternating with full UNet passes, and trains it on unrolled sampling trajectories.
Problem
Diffusion models are slow because each denoising step runs a computationally expensive UNet; the work asks which parts of the UNet can be approximated cheaply without hurting output quality for text-to-image generation and editing.
Approach
Low-resolution UNet features are robust to perturbation and determine semantic layout, so a lightweight adaptor can approximate them from the previous step's outputs (model-step distillation), alternated with full UNet passes (clock schedule), and trained on unrolled sampling trajectories from noise without an image dataset.
Key results
- On SD UNet with 8 DPM++ steps on MS-COCO 2017, Clockwork cuts TFLOPs from 10.8 to 7.3 (-32%) with FID 24.64 to 24.11 and CLIP 0.300 to 0.295.
- On Distilled Efficient UNet at 8 steps, Clockwork reduces FLOPs by 38% (4.7 to 2.9 TFLOPs) with CLIP drop of 0.6% and FID improvement of 5%.
- On Plug-and-Play image editing with default 1000 inversion steps, Clockwork saves 33% compute while improving CLIP and slightly degrading DINO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply Clockwork with a clock of 2 | SD UNet · MS-COCO 2017-5K validation, 512×512 | FID: 24.64 -> 24.11 | Helped |
| Apply Clockwork with a clock of 2 | SD UNet · MS-COCO 2017-5K validation, 512×512 | TFLOPs: 10.8 -> 7.3 (−32%) | Helped |
| Apply Clockwork with a clock of 2 | Efficient UNet · MS-COCO 2017-5K validation, 512×512 | FID: 24.22 -> 23.21 | Helped |
| Apply Clockwork with a clock of 2 | Efficient UNet · MS-COCO 2017-5K validation, 512×512 | TFLOPs: 9.5 -> 5.9 (−38%) | Helped |
| Apply Clockwork with a clock of 2 | Distilled Efficient UNet · MS-COCO 2017-5K validation, 512×512 | FID: 25.75 -> 24.45 | Helped |
| Apply Clockwork with a clock of 2 | Distilled Efficient UNet · MS-COCO 2017-5K validation, 512×512 | TFLOPs: 4.7 -> 2.9 (−38%) | Helped |
| Apply Clockwork with a clock of 2 | SD UNet · MS-COCO 2014-30K validation, 256×256 | FID: 12.77 -> 12.27 | Helped |
| Apply Clockwork with a clock of 2 | Efficient UNet · MS-COCO 2014-30K validation, 256×256 | FID: 12.33 -> 11.14 | Helped |
| Use consecutive adaptor passes instead of alternating with full UNet passes (ablation) | Clockwork with Efficient UNet · MS-COCO 2017-5K validation, 512×512, 8 steps | FID: 23.21 -> 28.07 | Hurt |
| Use consecutive adaptor passes at steps {3, 4, 5, 6} (ablation) | Clockwork with Efficient UNet · MS-COCO 2017-5K validation, 512×512, 8 steps | FID: 23.21 -> 33.10 | Hurt |
| Replace the ResNet adaptor with a UNet-like adaptor (ablation) | Clockwork with Efficient UNet · MS-COCO 2017-5K validation, 512×512, 8 steps | GFLOPs: 301 -> 324 | Hurt |
| Move the UNet cut-off from stage 1 to stage 2 (ablation) | Clockwork with Efficient UNet · MS-COCO 2017-5K validation, 512×512, 8 steps | FID: 23.21 -> 24.49 | Hurt |
Surprising findings
- Approximating early steps harms generation far more than approximating later steps (alternating clock on steps 3-6 gives FID 33.10 vs 23.21 for the default).
- Clockwork's FID sometimes improves over the baseline (e.g., Efficient UNet FID 24.22 to 23.21).
Limitations
- Clockwork is trained for a fixed operating point and is sensitive to drastic changes in scheduler or number of sampling steps.
- Demonstrated only on UNet-based diffusion models; applicability to ViT-based architectures is unclear.
- Reason why unrolled training outperforms regular distillation at low steps is not fully understood.
Experimental setup
Datasets: LAION-5B (50K captions) MS-COCO 2017 MS-COCO 2014 ImageNet-R-TI2I
Benchmarks: MS-COCO 2017 5K MS-COCO 2014 30K ImageNet-R-TI2I
Metrics: FID CLIP score TFLOPs latency DINO self-similarity
Baselines: Meng et al. classifier-free guidance distillation SnapFusion BK-SDM InstaFlow Plug-and-Play
Builds on: Stable Diffusion v1.5 DPM++ progressive step distillation classifier-free guidance distillation Plug-and-Play
Closest papers by meaning
- Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference (NeurIPS 2024 · similarity 0.89)
- InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation (ICLR 2024 · similarity 0.88)
- Denoising Diffusion Bridge Models (ICLR 2024 · similarity 0.88)
- DreamTime: An Improved Optimization Strategy for Diffusion-Guided 3D Generation (ICLR 2024 · similarity 0.88)
- Think while You Generate: Discrete Diffusion with Planned Denoising (ICLR 2025 · similarity 0.88)
- One Stone with Two Birds: A Null-Text-Null Frequency-Aware Diffusion Models for Text-Guided Image Inpainting (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.