Research map / Architectures and efficiency / Quantization
VQ4DiT: Efficient Post-Training Vector Quantization for Diffusion Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VQ4DiT, a zero-data, block-wise post-training vector quantization method that jointly calibrates codebooks and candidate assignment ratios for DiT weights, reaching near-lossless 3-bit and usable 2-bit DiT-XL/2 generation.
Problem
Deploying Diffusion Transformers (DiTs) on edge devices is impractical due to their large parameter counts; existing PTQ and classic VQ methods degrade badly at extremely low bit-widths, and traditional VQ calibrates only codebooks, not assignments.
Approach
For each weight sub-vector, keep a small set of candidate codewords (nearest by Euclidean distance) with learnable softmax ratios, reconstruct weights as a weighted average, and jointly calibrate codebooks and ratios with zero-data, block-wise MSE distillation from the floating-point model; the highest-ratio candidate becomes the final assignment.
Key results
- On ImageNet 256x256 with 3-bit weights and 50 steps, VQ4DiT reaches FID 11.91 (FP 6.72) versus 71.42 for GPTQ and 1.5e2 for Q-DiT.
- At 2-bit weights (250 steps, 256x256), VQ4DiT achieves FID 11.87 and IS 219.33 while RepQ-ViT, GPTQ and Q-DiT collapse (FID 2.8e2 to 3.1e2).
- Ablation at 2-bit shows candidate set length n=2 gives FID 12.42 vs 60.16 for n=1; n=3 gives slightly better FID 12.22 but n=2 is chosen.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VQ4DiT instead of GPTQ for 3-bit quantization | DiT XL/2 · ImageNet 256×256, 50 steps | FID: 71.42 -> 11.91 | Helped |
| Use VQ4DiT instead of GPTQ for 2-bit quantization | DiT XL/2 · ImageNet 256×256, 50 steps | FID: 3.1e2 -> 12.42 | Helped |
| Use VQ4DiT instead of GPTQ for 3-bit quantization | DiT XL/2 · ImageNet 512×512, 50 steps | FID: 94.35 -> 34.57 | Helped |
| Use VQ4DiT instead of GPTQ for 2-bit quantization | DiT XL/2 · ImageNet 512×512, 50 steps | FID: 3.2e2 -> 35.08 | Helped |
| Increase candidate assignment set length from n=1 to n=2 (ablation) | VQ4DiT · ImageNet 256×256, 2-bit, 50 steps | FID: 60.16 -> 12.42 | Helped |
| Increase candidate assignment set length from n=2 to n=3 (ablation) | VQ4DiT · ImageNet 256×256, 2-bit, 50 steps | FID: 12.42 -> 12.22 | Helped |
| Increase candidate assignment set length from n=3 to n=4 (ablation) | VQ4DiT · ImageNet 256×256, 2-bit, 50 steps | FID: 12.22 -> 14.06 | Hurt |
| Use CUDA vector quantization kernel | 3-bit VQ, DiT XL/2 · 256×256, 256 steps | inference time: 63s -> 22s | Helped |
Surprising findings
- Calibrating only the codebook with fine-tuning gave only slight gains over no fine-tuning for classic VQ on DiT, motivating joint assignment calibration.
- Increasing the candidate set length beyond 3 (n=4) degraded performance, suggesting excess candidates hinder calibration convergence.
Limitations
- Only DiT blocks are quantized; other components (e.g., embeddings, adaLN) are left unquantized for baseline consistency.
- Baselines were re-implemented by the authors for DiTs rather than taken from original releases.
- Evaluation is limited to DiT-XL/2 on ImageNet class-conditional generation.
Experimental setup
Datasets: ImageNet
Benchmarks: ImageNet 256x256 ImageNet 512x512
Metrics: FID sFID IS Precision model size (MB) inference time
Baselines: RepQ-ViT GPTQ Q-DiT uniform quantization classic VQ with codebook fine-tuning
Builds on: K-Means DiT GPTQ RepQ-ViT Q-DiT
Closest papers by meaning
- ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generation (ICLR 2025 · similarity 0.93)
- Q-DiT: Accurate Post-Training Quantization for Diffusion Transformers (CVPR 2025 · similarity 0.92)
- PTQ4DiT: Post-training Quantization for Diffusion Transformers (NeurIPS 2024 · similarity 0.92)
- Q-VDiT: Towards Accurate Quantization and Distillation of Video-Generation Diffusion Transformers (ICML 2025 · similarity 0.92)
- DiRotQ: Rotation-Aware Quantization for 4-bit Diffusion Transformers (NeurIPS 2026 · similarity 0.92)
- $\text{S}^2$Q-VDiT: Accurate Quantized Video Diffusion Transformer with Salient Data and Sparse Token Distillation (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.