Research map / Image generation / Fast diffusion sampling
BinaryDM: Accurate Weight Binarization for Efficient Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BinaryDM, a quantization-aware training method that binarizes diffusion-model weights using an Evolvable-Basis Binarizer and low-rank representation mimicking, reaching W1A4 with competitive FID on DDIM and LDM benchmarks.
Problem
Binarizing diffusion model weights to 1 bit causes severe representation loss and unstable QAT convergence, so binarized diffusion models collapse at low activation bit-widths.
Approach
An Evolvable-Basis Binarizer uses residual multi-basis binarization with learnable scalars during early training, regularized to collapse to a single binary basis, applied only to head and tail layers; Low-rank Representation Mimicking distills PCA-projected full-precision intermediate features into the binarized model.
Key results
- On LSUN-Bedrooms LDM-4 with W1A4, BinaryDM reaches FID 7.74 versus 10.87 for the binarized baseline.
- W1A4 BinaryDM reports 15.2x OPs and 29.2x model size savings relative to full precision.
- On LSUN-Churches LDM-8, W1A4 BinaryDM FID 9.91 vs baseline 12.98; the paper claims W1A4 beats W4A4 EfficientDM on FID by 4.43.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add EBB to vanilla binarization (ablation) | BinaryDM · LSUN-Bedrooms 256×256 W1A32 | FID: 8.43 to 7.39 | Helped |
| Add LRM after EBB (ablation) | BinaryDM · LSUN-Bedrooms 256×256 W1A32 | FID: 7.39 to 6.99 | Helped |
| Add EBB to vanilla binarization (ablation) | BinaryDM · LSUN-Bedrooms 256×256 W1A4 | FID: 10.87 to 8.53 | Helped |
| Add LRM after EBB (ablation) | BinaryDM · LSUN-Bedrooms 256×256 W1A4 | FID: 8.53 to 7.74 | Helped |
| Apply EBB to 6 head and tail blocks, not central blocks (ablation) | BinaryDM · LSUN-Bedrooms 256×256 W1A32 | FID: 8.02 to 6.99 | Helped |
| Increase EBB coverage to 12 head and tail blocks and 1 central block instead of selective 6 (ablation) | BinaryDM · LSUN-Bedrooms 256×256 W1A32 | FID: 6.99 to 7.10 | Hurt |
| Use regularization coefficient 9e-2 rather than no regularization (ablation) | EBB · LSUN-Bedrooms 256×256 W1A32 | FID: 8.01 to 6.99 | Helped |
| Extend EBB regularization phase from 10000 to 100000 iterations (ablation) | EBB · LSUN-Bedrooms 256×256 W1A32 | FID: 7.08 to 6.99 | Helped |
| Remove the sign function from EBB's second term (ablation) | EBB · LSUN-Bedrooms 256×256 W1A4 | FID: 7.74 to 8.44 | Hurt |
| Use LRM instead of direct feature MSE distillation (ablation) | BinaryDM with EBB · LSUN-Bedrooms 256×256 W1A32 | FID: 7.36 to 6.99 | Helped |
| Set LRM dimension reduction K to 8 rather than 4 (ablation) | BinaryDM with EBB and LRM · LSUN-Bedrooms 256×256 W1A32 | FID: 6.99 to 6.95 | Helped |
| Update the LRM projection matrix every 100 iterations instead of keeping it fixed (ablation) | BinaryDM with LRM · LSUN-Bedrooms 256×256 W1A32 | FID: 6.99 to 7.11 | Hurt |
Surprising findings
- Removing the sign function from the second EBB term gave worse results (FID 8.44 vs 7.74), since it hinders evolution to a single basis.
- Fixing the LRM projection matrix after the first batch outperformed periodically updating it (FID 6.99 vs 7.11 at update every 100 iterations).
Limitations
- Training overhead is higher than post-training quantization methods (training hours comparable to Q-Diffusion calibration).
- Hardware speedups are limited by lack of native W1A4 support in deployment libraries; measured speedup is below theoretical OPs.
Experimental setup
Datasets: CIFAR-10 LSUN-Bedrooms LSUN-Churches FFHQ ImageNet
Benchmarks: CIFAR-10 LSUN-Bedrooms LSUN-Churches FFHQ ImageNet
Metrics: FID sFID IS Precision Recall OPs model size
Baselines: LSQ Q-Diffusion EfficientDM ReActNet INSTA-BNN Q-DM TDQ BI-DiffSR Baseline binarized DM
Builds on: LSQ ReActNet Q-Diffusion EfficientDM DDIM
Closest papers by meaning
- BiDM: Pushing the Limit of Quantization for Diffusion Models (NeurIPS 2024 · similarity 0.93)
- EfficientDM: Efficient Quantization-Aware Fine-Tuning of Low-Bit Diffusion Models (ICLR 2024 · similarity 0.90)
- MPQ-DM: Mixed Precision Quantization for Extremely Low Bit Diffusion Models (AAAI 2025 · similarity 0.90)
- Binarized Diffusion Model for Image Super-Resolution (NeurIPS 2024 · similarity 0.88)
- BitsFusion: 1.99 bits Weight Quantization of Diffusion Model (NeurIPS 2024 · similarity 0.87)
- TCAQ-DM: Timestep-Channel Adaptive Quantization for Diffusion Models (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.