Research map / Image generation / Fast diffusion sampling
ALTER: All-in-One Layer Pruning and Temporal Expert Routing for Efficient Diffusion Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ALTER, a single-stage framework where a trainable hypernetwork generates layer-pruning masks and routes diffusion timesteps to pruned temporal experts while fine-tuning SD-2.1's UNet.
Problem
Diffusion models are expensive at inference due to iterative denoising; existing structured layer pruning is static or sample-wise, and the pruning-then-fine-tuning pipeline misaligns pruning decisions with final weights.
Approach
Jointly optimize layer pruning, timestep-conditioned expert routing (Gumbel-based straight-through hypernetwork), and UNet fine-tuning in a single alternating stage, so each pruned expert serves a range of timesteps and co-adapts with the shared backbone.
Key results
- ALTER (0.65) at 20 steps reaches FID-5K 25.25 with 9.89T MACs and 3.64x speedup vs 50-step SDv2.1 (FID-5K 27.29, 38.04T MACs) on MS-COCO 2017.
- On MS-COCO 2014 at 25 steps, ALTER (0.65) achieves FID 13.70 and CLIP 32.17 versus SDv2.1's 14.29 and 32.17 at similar MACs (899.7G vs 1384.2G).
- Ablation on CC3M: single-stage joint training (FID 17.14) beats the two-stage variant (FID 17.65) and the static pruning variant (FID 19.03).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace static global pruning with multiple experts assigned to fixed timestep intervals (ablation) | ALTER · CC3M | FID: 19.03 -> 17.91 | Helped |
| use a learnable timestep router instead of fixed expert intervals (ablation) | ALTER · CC3M | FID: 17.91 -> 17.14 | Helped |
| use a learnable timestep router instead of fixed expert intervals (ablation) | ALTER · MS-COCO | FID: 14.13 -> 13.89 | Helped |
| use single-stage joint optimization instead of two-stage optimization (ablation) | ALTER · CC3M and MS-COCO | FID: 17.65 -> 17.14; 14.45 -> 13.89 | Helped |
| reduce target sparsity ratio from 0.65 to 0.60 (ablation) | ALTER · CC3M and MS-COCO, 20 steps | FID: 17.14 -> 17.87; 13.89 -> 14.24 | Hurt |
| reduce target sparsity ratio from 0.65 to 0.55 (ablation) | ALTER · CC3M and MS-COCO, 20 steps | FID: 17.14 -> 18.27; 13.89 -> 14.59 | Hurt |
| reduce target sparsity ratio from 0.65 to 0.50 (ablation) | ALTER · CC3M and MS-COCO, 20 steps | FID: 17.14 -> 18.99; 13.89 -> 14.70 | Hurt |
| add output-level knowledge distillation to the denoising loss (ablation) | Static pruning variant · CC3M | FID: 23.76 -> 20.15 | Helped |
| add feature-level knowledge distillation to denoising loss plus output KD (ablation) | Static pruning variant · CC3M | FID: 20.15 -> 19.03 | Helped |
| increase the number of temporal experts from fewer than 10 to 10 (ablation) | ALTER · CC3M and MS-COCO | FID | Helped |
| increase the number of temporal experts from 10 to 15 (ablation) | ALTER · CC3M and MS-COCO | FID | No clear effect |
Surprising findings
- ALTER at 20 steps slightly outperforms the 25-step and 50-step unpruned SDv2.1 on FID and CLIP in several settings.
- Static pruning with distillation losses alone degrades quality noticeably versus the full ALTER approach, while the sample-wise APTP (0.66) degrades severely (CC3M FID 60.04).
Limitations
- A final fine-tuning stage after T_end is still needed to recover performance.
- Increasing experts beyond 10 (to 15) did not improve results and sometimes degraded them.
- Compared on relatively few baselines; evaluation limited to SD-v2.1 and COCO/CC3M FID-based metrics.
Experimental setup
Datasets: LAION-Aesthetics V2 (6.5+) subset
Benchmarks: MS-COCO 2014 validation CC3M validation MS-COCO 2017 5k validation
Metrics: FID CMMD CLIP score MACs latency speedup
Baselines: BK-SDM-v2 (Base) BK-SDM-v2 (Small) APTP DiP-GO SDv2.1 (50 steps) SDv2.1 (20 steps)
Builds on: APTP BK-SDM Gumbel-Softmax classifier-free guidance
Closest papers by meaning
- DiP-GO: A Diffusion Pruner via Few-step Gradient Optimization (NeurIPS 2024 · similarity 0.92)
- Attention-Driven Training-Free Efficiency Enhancement of Diffusion Models (CVPR 2024 · similarity 0.91)
- Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference (NeurIPS 2024 · similarity 0.90)
- Analyzing and Improving the Training Dynamics of Diffusion Models (CVPR 2024 · similarity 0.90)
- Rethinking Cross-Layer Information Routing in Diffusion Transformer (NeurIPS 2026 · similarity 0.90)
- Dual-Rate Diffusion: Accelerating diffusion models with an interleaved heavy-light network (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.