Research map / Image generation / Fast diffusion sampling
BitsFusion: 1.99 bits Weight Quantization of Diffusion Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Quantizes the SD-v1.5 UNet to an average 1.99 bits via mixed-precision layer assignment, balanced integer levels, alternating scale initialization, and two-stage distillation with error-aware timestep sampling, yielding a 7.9x smaller model that matches or beats FP32 SD-v1.5 on CLIP, TIFA, GenEval, and human preference.
Problem
Stable Diffusion v1.5's UNet is large (1.72 GB in FP16), and prior diffusion quantization work mostly targets 4-bit-and-above weights on small-scale models, leaving extremely low-bit quantization of large text-to-image diffusion models underexplored.
Approach
Combine a per-layer sensitivity analysis (MSE scaled by layer size plus CLIP score drop) to assign mixed bit-widths with a balanced integer level set (log2(2^n+1) bits), alternating-optimization scale initialization, caching of time embeddings, and a two-stage QAT pipeline (CFG-aware distillation with feature loss and Beta-sampled timesteps focused on high-error steps, then noise-prediction fine-t
Key results
- The 1.99-bit UNet (219 MB, 7.9x smaller than the 1.72 GB FP16 UNet) outperforms SD-v1.5 on CLIP score by 0.002-0.003 across CFG scales 2.5-9.5 on MS-COCO.
- On 1K PartiPrompts CLIP score, the model reaches 0.3212 at CFG 7.5 versus 0.3175 for SD-v1.5, and beats LSQ (2-bit, 0.2849), Q-Diffusion (4-bit, 0.3137), EfficientDM (2-bit, 0.2918) and Apple-MBP (2-bit, 0.3023).
- In human evaluation on PartiPrompts, users preferred BitsFusion over SD-v1.5 in 54.4% of comparisons.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Quantize the SD-v1.5 UNet to 1.99 bits with BitsFusion | Stable Diffusion v1.5 · 1K PartiPrompts | CLIP score: 0.3175 -> 0.3212 | Helped |
| Add a balance integer (ablation) | 2-bit QAT-Base · 1K PartiPrompts | Average CLIP score: 0.2797 -> 0.3054 | Helped |
| Add alternating optimization for scaling-factor initialization (ablation) | BitsFusion + Balance · 1K PartiPrompts | Average CLIP score: 0.3054 -> 0.3100 | Helped |
| Add mixed-precision quantization and time-embedding caching (ablation) | BitsFusion + Alternating Opt. · 1K PartiPrompts | Average CLIP score: 0.3100 -> 0.3118 | Helped |
| Add feature distillation (ablation) | BitsFusion + Mixed/Caching · 1K PartiPrompts | Average CLIP score: 0.3118 -> 0.3142 | Helped |
| Add quantization error-aware time-step sampling (ablation) | BitsFusion + Feat Dist. · 1K PartiPrompts | Average CLIP score: +0.0014 | Helped |
| Add Stage-II noise-prediction fine-tuning (ablation) | BitsFusion + Time Sampling · 1K PartiPrompts | Average CLIP score: +0.0027 | Helped |
| Use parameter-size factor eta 0.3 rather than eta 0 (ablation) | BitsFusion mixed-precision strategy · 1K PartiPrompts | CLIP score: 0.3155 -> 0.3181 | Helped |
| Set feature-distillation weight lambda to 1 rather than 0.01 (ablation) | BitsFusion · 1K PartiPrompts | CLIP score: 0.3181 -> 0.3164 | Hurt |
| Set feature-distillation weight lambda to 0.1 rather than 0.01 (ablation) | BitsFusion · 1K PartiPrompts | CLIP score: 0.3181 -> 0.3159 | Hurt |
| Set time-sampling Beta distribution alpha to 3.0 rather than 1.5 (ablation) | BitsFusion · 1K PartiPrompts | CLIP score: 0.3169 -> 0.3181 | Helped |
| Set time-sampling Beta distribution alpha to 3.0 rather than 2.0 (ablation) | BitsFusion · 1K PartiPrompts | CLIP score: 0.3173 -> 0.3181 | Helped |
Surprising findings
- The quantized model outperforms the full-precision SD-v1.5 on CLIP score and human preference despite using 1.99 bits.
- FID of the Stage-II model is higher than both Stage-I and SD-v1.5, even though users prefer it, so the authors argue FID does not reflect human preference here.
Limitations
- Only the UNet is compressed; the VAE and CLIP text encoder are not quantized.
- Only weight quantization is studied; activation quantization is left for future work.
- Training uses an internal dataset that is not released, limiting reproducibility.
Experimental setup
Datasets: internal dataset MS-COCO 2014 PartiPrompts TIFA GenEval
Benchmarks: MS-COCO 2014 30K validation (CLIP score, FID) TIFA GenEval PartiPrompts (CLIP score, human evaluation)
Metrics: CLIP score FID TIFA score GenEval score human preference rate
Baselines: LSQ Q-Diffusion EfficientDM Apple-MBP SD-v1.5
Builds on: LSQ Q-Diffusion EfficientDM classifier-free guidance
Closest papers by meaning
- Bridging the Gap between Gaussian Diffusion Models and Universal Quantization for Image Compression (CVPR 2025 · similarity 0.91)
- Qua2SeDiMo: Quantifiable Quantization Sensitivity of Diffusion Models (AAAI 2025 · similarity 0.90)
- BiDM: Pushing the Limit of Quantization for Diffusion Models (NeurIPS 2024 · similarity 0.90)
- EfficientDM: Efficient Quantization-Aware Fine-Tuning of Low-Bit Diffusion Models (ICLR 2024 · similarity 0.90)
- Optimizing Quantized Diffusion Models via Distillation with Cross-Timestep Error Correction (AAAI 2025 · similarity 0.89)
- MPQ-DM: Mixed Precision Quantization for Extremely Low Bit Diffusion Models (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.