Research map / Image generation / Fast diffusion sampling
FreqTS: Frequency-Aware Token Selection for Accelerating Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
FreqTS accelerates diffusion models without retraining by amplitude-based frequency sorting of tokens, selective retention of low-frequency tokens, and Bayesian-optimized per-step selection ratios.
Problem
Diffusion models are computationally expensive due to sequential denoising and large model size, and they over-emphasize low-frequency features, giving sub-optimal quality.
Approach
Sort latent tokens by frequency-domain amplitude into high- and low-frequency subsets, keep all high-frequency tokens and a Beta-distributed fraction of low-frequency tokens, and tune the per-timestep retention parameters with Bayesian optimization, with no retraining.
Key results
- FreqTS reduces PixArt-Alpha inference latency by 38.4% (61.5s to 37.9s) with FID 35.8 vs. baseline 38.7.
- FID on MS-COCO rises modestly for SD-1.5 (23.9 to 20.8 reported as increase), SD-2.1 (22.6 to 19.9) and SD-XL (24.6 to 22.7) with reduced MACs.
- Claims at least 2.3x speedup without retraining; with Bayesian selection, latency 2.914 and FID 6.20 vs. SDXL-Turbo FID 22.58.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply FreqTS frequency-aware token selection | SD-1.5 · MS-COCO, zero-shot 10K-FID | FID: 23.9 -> 20.8 | Helped |
| Apply FreqTS frequency-aware token selection | SD-2.1 · MS-COCO, zero-shot 10K-FID | FID: 22.6 -> 19.9 | Helped |
| Apply FreqTS frequency-aware token selection | SD-XL · MS-COCO, zero-shot 10K-FID | FID: 24.6 -> 22.7 | Helped |
| Apply FreqTS frequency-aware token selection | PixArt-Alpha · PixArt-Alpha | latency: 61.5s -> 37.9s | Helped |
| Apply FreqTS frequency-aware token selection | PixArt-Alpha · PixArt-Alpha | FID: 38.7 -> 35.8 | Helped |
| Raise the token sort threshold from 40% to 70% (ablation) | FreqTS · Figure 4 comparison | latency: 4.183 -> 2.492 | Helped |
| Raise the token sort threshold from 40% to 70% (ablation) | FreqTS · Figure 4 comparison | FID: 6.40 -> 7.83 | Hurt |
| Add Bayesian selection strategy optimization (ablation) | FreqTS · Figure 4 comparison | FID: 7.83 -> 6.20 | Helped |
| Add Bayesian selection strategy optimization (ablation) | FreqTS · Figure 4 comparison | latency: 2.492 -> 2.914 | Hurt |
Limitations
- Acceleration claims are reported with inconsistent FID and latency numbers across tables and text, making comparisons hard to verify.
- Method description of the Bayesian search objective is vague (acquisition function and performance metric not concretely specified).
Experimental setup
Datasets: MS-COCO
Benchmarks: MS-COCO
Metrics: FID latency MACs parameter count
Baselines: SDXL-Turbo SDXL-Lightning DDIM DPM-Solver++ DeepCache LCM
Builds on: DDPM DDIM DeepCache Token Merging
Closest papers by meaning
- EfficientDM: Efficient Quantization-Aware Fine-Tuning of Low-Bit Diffusion Models (ICLR 2024 · similarity 0.90)
- FreqPrior: Improving Video Diffusion Models with Frequency Filtering Gaussian Noise (ICLR 2025 · similarity 0.89)
- Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference (NeurIPS 2024 · similarity 0.89)
- FAM Diffusion: Frequency and Attention Modulation for High-Resolution Image Generation with Stable Diffusion (CVPR 2025 · similarity 0.89)
- Boosting Diffusion Models with Moving Average Sampling in Frequency Domain (CVPR 2024 · similarity 0.89)
- A Simple Early Exiting Framework for Accelerated Sampling in Diffusion Models (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.