Research map / Image generation / Fast diffusion sampling

Training-Free and Hardware-Friendly Acceleration for Diffusion Models via Similarity-based Token Pruning

AAAI 2025 · Image generation › Fast diffusion sampling · approach: Training-free diffusion acceleration · 10 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SiTo, a training-free, data-free similarity-based token pruning method that speeds up Stable Diffusion inference (about 1.9x on ImageNet, 1.75x on COCO30K) while keeping or improving FID.

Problem

Diffusion models are computationally expensive at inference, and prior token-reduction methods for diffusion (ToMeSD, AT-EDM) either are designed for classification, have unpredictable runtime, or are hard to batch on hardware.

Approach

Select base tokens uniformly across spatial patches using cosine-similarity sums perturbed by Gaussian noise, prune the tokens most similar to base tokens, run the layer only on unpruned tokens, and recover pruned tokens by copying their most similar base token; uses only cheap non-parametric operations.

Key results

• On SD v1.5 with ImageNet, SiTo-e reaches 1.90x speedup and 2.70x memory compression with FID 26.31 vs 27.64 original.
• On COCO30K with SD v1.5, SiTo-d achieves FID 11.17 (vs 12.32 original) at 1.75x speedup, while ToMeSD-d reaches 12.15 FID at 1.71x.
• Ablation shows noise-augmented base token selection (VI) improves FID to 26.19 vs 26.40 for the deterministic patch-local variant (V) at similar speedup.

Measured effects reported in the paper

Surprising findings

• Pruning-based SiTo achieves lower FID than the original uncompressed Stable Diffusion model on ImageNet and COCO30K.
• Pruning consistently beats average merging at the same token reduction ratio, contrary to the usual merge-based intuition.

Limitations

• Gains are evaluated only on Stable Diffusion v1.5 and v2 with PLMS 50 steps; generalization to other diffusion architectures is not shown.
• Applying SiTo to cross-attention or FFN layers gives minimal speedup while hurting quality, so benefits are concentrated in self-attention of outer blocks.
• Speed and quality measured on a single GPU setup; FID on 2,000 ImageNet and 30K COCO images is a coarse quality measure.

Experimental setup

Datasets: ImageNet-1k COCO30K

Benchmarks: ImageNet-1k COCO30K

Metrics: FID latency speedup ratio memory usage CLIP score

Baselines: ToMeSD AT-EDM SD v1.5 original SD v2 original

Builds on: ToMeSD Token Merging DDIM PLMS

Closest papers by meaning

• DiP-GO: A Diffusion Pruner via Few-step Gradient Optimization (NeurIPS 2024 · similarity 0.90)
• ALTER: All-in-One Layer Pruning and Temporal Expert Routing for Efficient Diffusion Generation (NeurIPS 2025 · similarity 0.89)
• Flash Diffusion: Accelerating Any Conditional Diffusion Model for Few Steps Image Generation (AAAI 2025 · similarity 0.88)
• Attention-Driven Training-Free Efficiency Enhancement of Diffusion Models (CVPR 2024 · similarity 0.88)
• DeepCache: Accelerating Diffusion Models for Free (CVPR 2024 · similarity 0.88)
• ToMA: Token Merge with Attention for Diffusion Models (ICML 2025 · similarity 0.88)