Research map / Architectures and efficiency / Quantization
EfficientDM: Efficient Quantization-Aware Fine-Tuning of Low-Bit Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EfficientDM, a data-free, parameter-efficient quantization-aware fine-tuning framework that merges quantization-aware LoRA into weights to get QAT-level quality of low-bit diffusion models at PTQ-level cost.
Problem
Post-training quantization of diffusion models degrades badly at low bit-widths (e.g., W4A4), while quantization-aware training recovers quality but needs training data and large compute and memory.
Approach
Quantization-aware LoRA (QALoRA) merges low-rank updates into the FP weights and jointly quantizes them, so no full-precision adapter remains at inference. Fine-tuning distills the FP model's noise predictions into the quantized model using only self-generated denoising trajectories. Scale-aware gradient scaling and per-timestep learned activation step sizes (TALSQ) address learning and distributi
Key results
- On ImageNet 256x256 LDM-4 (DDIM, 20 steps), W4A4 reaches FID 6.17 and sFID 7.75, where PTQ baselines fail to generate images.
- On CIFAR-10 W4A8, FID 3.80 beats the QAT method TDQ (4.13) and matches PTQ-level time (0.97 vs 0.95 GPU hours).
- Compared with QAT on LSUN-Churches, the method needs 1.14 GPU hours versus 18.5 for QAT, a 16.2x speed-up.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use EfficientDM rather than TDQ at W4A8 | DDIM · CIFAR-10 32×32 | FID: 4.13 -> 3.80 | Helped |
| use EfficientDM rather than PTQD at W4A8 | LDM-4 with DDIM 20 steps · ImageNet 256×256 | IS: 344.72 -> 353.83 | Helped |
| add scale-aware LoRA optimization to QALoRA (ablation) | LDM-4 · ImageNet 256×256, W4A4 | FID: 11.42 -> 9.55 | Helped |
| add scale-aware LoRA optimization to QALoRA (ablation) | LDM-4 · ImageNet 256×256, W4A4 | sFID: 24.18 -> 13.23 | Helped |
| add TALSQ to QALoRA with scale-aware LoRA optimization (ablation) | LDM-4 · ImageNet 256×256, W4A4 | FID: 9.55 -> 6.17 | Helped |
| add TALSQ to QALoRA with scale-aware LoRA optimization (ablation) | LDM-4 · ImageNet 256×256, W4A4 | sFID: 13.23 -> 7.75 | Helped |
| replace learned temporal scales with PTQ time-step-wise quantization (ablation) | QALoRA with scale-aware LoRA optimization · ImageNet 256×256, W4A4 | FID: 6.17 -> 7.81 | Hurt |
| set LoRA rank to 8 rather than 32 (ablation) | LDM-8 · LSUN-Churches, W6A6 | FID: 6.29 -> 10.53 | Hurt |
| set LoRA rank to 16 rather than 32 (ablation) | LDM-8 · LSUN-Churches, W6A6 | FID: 6.29 -> 8.06 | Hurt |
| increase LoRA rank from 32 to 64 (ablation) | LDM-8 · LSUN-Churches, W6A6 | FID: 6.29 -> 6.20 | Helped |
| use EfficientDM rather than PTQ | DDIM · CIFAR-10, W4A4 | FID: 181.05 -> 10.48 | Helped |
| use EfficientDM rather than QAT | DDIM · CIFAR-10, W4A4 | FID: 7.30 -> 10.48 | Hurt |
Surprising findings
- PTQD fails to denoise at W4A4 on ImageNet (FID 259.73), while the proposed method keeps FID 6.17 at W4A4 and even reaches 2-bit weights with sFID under 8.
Limitations
- Gradient descent optimization of QALoRA needs more GPU memory than PTQ methods (19.8 GB on ImageNet vs about 12 GB for PTQ).
- Not yet evaluated on video or 3D diffusion models.
Experimental setup
Datasets: CIFAR-10 ImageNet 256x256 LSUN-Bedrooms LSUN-Churches
Benchmarks: CIFAR-10 ImageNet 256x256 LSUN-Bedrooms LSUN-Churches
Metrics: IS FID sFID Precision GPU time GPU memory
Baselines: PTQ4DM Q-Diffusion PTQD LSQ TDQ ADP-DM
Builds on: LoRA QLoRA LSQ Q-Diffusion PTQD
Closest papers by meaning
- Q-DiT: Accurate Post-Training Quantization for Diffusion Transformers (CVPR 2025 · similarity 0.92)
- Qua2SeDiMo: Quantifiable Quantization Sensitivity of Diffusion Models (AAAI 2025 · similarity 0.92)
- TCAQ-DM: Timestep-Channel Adaptive Quantization for Diffusion Models (AAAI 2025 · similarity 0.92)
- MPQ-DM: Mixed Precision Quantization for Extremely Low Bit Diffusion Models (AAAI 2025 · similarity 0.92)
- Towards Accurate Post-training Quantization for Diffusion Models (CVPR 2024 · similarity 0.91)
- SVDQuant: Absorbing Outliers by Low-Rank Component for 4-Bit Diffusion Models (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.