Research map / Architectures and efficiency / Quantization
Pioneering 4-Bit FP Quantization for Diffusion Models: Mixup-Sign Quantization and Timestep-Aware Fine-Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MSFP, a 4-bit floating-point quantization framework for diffusion models that mixes unsigned FP with zero point (for asymmetric activations) and signed FP, plus timestep-aware LoRA routing and denoising-factor loss alignment for fine-tuning.
Problem
4-bit quantization of diffusion models (DDIM and LDM) with fine-tuning remains inconsistent; signed FP quantization handles asymmetric post-SiLU activations poorly, a single LoRA is suboptimal across timesteps, and the MSE fine-tuning loss misaligns with actual quantization-induced degradation.
Approach
Use unsigned FP quantization with a learned zero point for asymmetric activation layers and signed FP for normal layers (mixup-sign), fine-tune with multiple LoRAs routed per timestep by a shared learnable router, and weight the noise-prediction loss by the denoising factor gamma_t.
Key results
- On CIFAR-10 with DDIM at 4/4 bits, FID 6.02 (h=2) versus 38.40 for EfficientDM and full precision 4.26.
- On LSUN-Church LDM-8 at 4/4 bits, FID 8.81 versus 18.40 for EfficientDM (full precision 4.06).
- Ablation on CelebA at 4/4 bits: combining MSFP, TALoRA and DFA reduces FID from 16.02 baseline to 7.69.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed 4-bit FP quantization and fine-tuning method | DDIM on CIFAR-10 · CIFAR-10 32x32, DDIM, 100 steps | FID: 38.40 -> 6.02 | Helped |
| Use the proposed 4-bit FP quantization and fine-tuning method | LDM-4 on LSUN-Bedroom · LSUN-Bedroom 256x256, 100 steps | FID: 36.36 -> 12.21 | Helped |
| Use the proposed 4-bit FP quantization and fine-tuning method | LDM-8 on LSUN-Church · LSUN-Church 256x256, 100 steps | FID: 13.03 -> 8.81 | Helped |
| Use the proposed 4-bit method | LDM-4 on ImageNet · ImageNet 256x256, 20 steps | sFID: 14.42 -> 7.42 | Helped |
| Add MSFP (ablation) | 4-bit FP diffusion model on CelebA · CelebA | FID: 16.02 -> 9.60 | Helped |
| Add TALoRA (ablation) | 4-bit FP diffusion model on CelebA · CelebA | FID: 16.02 -> 10.66 | Helped |
| Add DFA (ablation) | MSFP + TALoRA on the 4-bit FP diffusion model · CelebA | FID: 8.79 -> 7.69 | Helped |
| Allocate two LoRAs by splitting timesteps in half instead of using one LoRA (ablation) | 4-bit quantized DDIM · CelebA | FID: 19.41 -> 17.07 | Helped |
| Randomly allocate two LoRAs across timesteps instead of splitting steps in half (ablation) | 4-bit quantized DDIM · CelebA | FID: 17.07 -> 41.96 | Hurt |
Surprising findings
- On ImageNet, FID improved as bit-width decreased, contrary to expectation, so the authors rely on sFID and IS instead.
- Random allocation between two LoRAs performed much worse (FID 41.96) than a single LoRA (19.41).
Limitations
- Some FP-quantized baselines fail (N/A) at 4-bit, and results depend on the chosen LoRA Hub size h.
- FID is not reliable as a metric on ImageNet conditional generation according to the authors.
Experimental setup
Datasets: CIFAR-10 CelebA LSUN-Bedroom LSUN-Church ImageNet
Benchmarks: CIFAR-10 CelebA LSUN-Bedroom LSUN-Church ImageNet
Metrics: FID IS sFID
Baselines: Q-Diffusion EDA-DM EfficientDM QuEST
Builds on: EfficientDM Q-Diffusion LoRA search-based FP quantization DDIM
Closest papers by meaning
- EfficientDM: Efficient Quantization-Aware Fine-Tuning of Low-Bit Diffusion Models (ICLR 2024 · similarity 0.91)
- MPQ-DM: Mixed Precision Quantization for Extremely Low Bit Diffusion Models (AAAI 2025 · similarity 0.91)
- Qua2SeDiMo: Quantifiable Quantization Sensitivity of Diffusion Models (AAAI 2025 · similarity 0.90)
- AccuQuant: Simulating Multiple Denoising Steps for Quantizing Diffusion Models (NeurIPS 2025 · similarity 0.89)
- TCAQ-DM: Timestep-Channel Adaptive Quantization for Diffusion Models (AAAI 2025 · similarity 0.89)
- Q-DiT: Accurate Post-Training Quantization for Diffusion Transformers (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.