Research map / Image generation / Fast diffusion sampling
Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes encoder propagation for diffusion UNets/DiTs: reuse cached encoder features across adjacent time-steps and run decoder steps in parallel, plus prior noise injection, giving training-free inference speedups.
Problem
Diffusion model inference is slow due to sequential denoising; distillation approaches need costly retraining, and prior caching methods (DeepCache, CacheMe) remain sequential.
Approach
Empirically, UNet encoder features change minimally across adjacent time-steps while decoder features vary substantially; so the encoder is skipped at non-key time-steps, its cached features are reused by the decoder, and the decoder computations for those time-steps are run in parallel.
Key results
- On Stable Diffusion with DDIM (50 steps), encoder propagation with parallel decoding cuts sampling time from 2.42 to 1.42 s/image (41% reduction) with FID 21.08 vs 21.75 for the baseline.
- On DiT at 256 resolution, sampling time drops from 5.13 to 3.62 s/image (29%) with FID 2.31 vs 2.27.
- On DeepFloyd-IF with DDPM (225 steps), parallel propagation reduces time from 34.55 to 26.27 s/image (24%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply encoder propagation | Stable Diffusion with DDIM · MS-COCO 2017 10K subset | s/image: 2.42 -> 1.42 | Helped |
| Apply encoder propagation | Stable Diffusion with DDIM · MS-COCO 2017 10K subset | FID: 21.75 -> 21.08 | Helped |
| Add encoder propagation | Stable Diffusion with DPM-Solver++ · MS-COCO 2017 10K subset | FID: 20.51 -> 20.76 | Hurt |
| Apply encoder propagation | DeepFloyd-IF with DDPM · Text-to-image | s/image: 34.55 -> 26.27 | Helped |
| Apply encoder propagation | DiT · ImageNet 256 | s/image: 5.13 -> 3.62 | Helped |
| Use the selected non-uniform key time-steps rather than uniform propagation (ablation) | Stable Diffusion encoder propagation · MS-COCO 2017 10K subset | FID: 21.55 -> 21.08; 21.54 -> 21.08; 24.61 -> 21.08 | Mixed |
| Use non-uniform key time-steps II instead of the chosen key time-steps (ablation) | Stable Diffusion encoder propagation · MS-COCO 2017 10K subset | FID: 21.08 -> 35.25 | Hurt |
| Add prior zT noise injection (ablation) | Stable Diffusion encoder propagation · MS-COCO 2017 10K subset | FID: 21.71 -> 21.08 | Helped |
| Add prior zT noise injection (ablation) | Stable Diffusion encoder propagation · MS-COCO 2017 10K subset | Clipscore: 0.779 -> 0.783 | Helped |
Surprising findings
- Propagating decoder features instead of encoder features causes missing objects in generated images (e.g., glasses omitted), indicating semantics reside in decoder features.
- Simply lowering DDIM step count to match speed degrades quality much more than the proposed method at similar cost.
Limitations
- Quality degrades when using very few sampling steps (e.g., 5).
- Parallel propagation increases memory consumption (about 51% more on SD).
- Not combined with or evaluated on distillation-based methods.
Experimental setup
Datasets: MS-COCO 2017 validation ImageNet
Benchmarks: MS-COCO 2017 10K ImageNet 50K
Metrics: FID CLIPScore sFID IS Precision Recall GFLOPs/image s/image user preference
Baselines: DDIM DPM-Solver DPM-Solver++ DDPM ToMe DeepCache CacheMe Text2Video-zero VideoFusion ControlNet Dreambooth Custom Diffusion
Builds on: DDIM DPM-Solver ToMe DeepCache
Closest papers by meaning
- One-Way Ticket: Time-Independent Unified Encoder for Distilling Text-to-Image Diffusion Models (CVPR 2025 · similarity 0.93)
- Encoder-Decoder Diffusion Language Models for Efficient Training and Inference (NeurIPS 2025 · similarity 0.93)
- Dual-Rate Diffusion: Accelerating diffusion models with an interleaved heavy-light network (NeurIPS 2026 · similarity 0.92)
- Plug-and-Play Diffusion Distillation (CVPR 2024 · similarity 0.92)
- A Simple Early Exiting Framework for Accelerated Sampling in Diffusion Models (ICML 2024 · similarity 0.91)
- Diffusion Model is Effectively Its Own Teacher (CVPR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.