Research map / Image generation / Fast diffusion sampling

Modulated Diffusion: Accelerating Generative Modeling with Modulated Quantization

ICML 2025 · Image generation › Fast diffusion sampling · approach: Training-free diffusion acceleration · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MoDiff, which quantizes temporal differences of linear-layer activations across diffusion steps with error compensation, enabling PTQ activation quantization down to about 3 bits on CIFAR-10 and LSUN without loss of quality.

Problem

Diffusion sampling is computationally expensive; existing caching methods accumulate reuse errors and PTQ methods struggle to quantize activations below 8 bits due to outliers and varying ranges across time steps.

Approach

Reformulate each linear layer's computation across time steps as incremental updates from the previous step's output, quantize the temporal activation difference (which has a smaller, more concentrated range), and feed back the quantization residual into the next step so errors do not accumulate. The method is agnostic to the underlying PTQ quantizer and sampler.

Key results

• On CIFAR-10 with DDIM, LCQ+MoDiff at 8/3 (W/A) achieves FID 4.14 and sFID 4.38, matching the full-precision-activation model (FID 4.24), while LCQ alone reaches FID 143.39.
• On LSUN-Churches with LDM-8 at 8/3, Q-Diff+MoDiff reaches FID 5.40 versus 367.51 for Q-Diff.
• On ImageNet with DiT-XL/2 at 8/6, PTQ4DiT+MoDiff gets FID 54.74 versus 200.26 for PTQ4DiT alone.

Measured effects reported in the paper

Surprising findings

• Naive caching-style error accumulation grows with reuse frequency; without error compensation, LCQ+MoDiff's error becomes significant below 6-bit activations, while error compensation keeps error small even at 3 bits.
• Using DDPM instead of DDIM reduces MoDiff's gains, since DDPM's injected noise increases adjacent-step feature differences and residual magnitudes.

Limitations

• Reports only theoretical bit-operation counts, not real running-time speedups; hardware implementation left to future work.
• Requires extra memory to store intermediate activations and outputs per layer, and needs bias-free layers.
• Low-bit activation quality still degrades on higher-resolution LDMs (e.g., 3-bit on Churches, 4-bit on Bedrooms with 8-bit weights).

Experimental setup

Datasets: CIFAR-10 LSUN-Bedrooms LSUN-Churches MS-COCO 2014 ImageNet

Benchmarks: CIFAR-10 LSUN-Churches LSUN-Bedrooms MS-COCO 2014 ImageNet 256x256

Metrics: IS FID sFID precision recall GBOPs

Baselines: Q-Diffusion LCQ Q-Diff PTQ4DiT MixDQ PTQD DeepCache LTQ

Builds on: Q-Diffusion DeepCache DDIM BRECQ PTQD

Closest papers by meaning

• Towards Accurate Post-training Quantization for Diffusion Models (CVPR 2024 · similarity 0.90)
• CacheQuant: Comprehensively Accelerated Diffusion Models (CVPR 2025 · similarity 0.90)
• Optimizing Quantized Diffusion Models via Distillation with Cross-Timestep Error Correction (AAAI 2025 · similarity 0.89)
• EfficientDM: Efficient Quantization-Aware Fine-Tuning of Low-Bit Diffusion Models (ICLR 2024 · similarity 0.89)
• AccuQuant: Simulating Multiple Denoising Steps for Quantizing Diffusion Models (NeurIPS 2025 · similarity 0.89)
• TCAQ-DM: Timestep-Channel Adaptive Quantization for Diffusion Models (AAAI 2025 · similarity 0.89)