Research map / Image generation / Fast diffusion sampling

ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generation

ICLR 2025 · Image generation › Fast diffusion sampling · approach: Training-free diffusion acceleration · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ViDiT-Q, a post-training quantization scheme for video and image diffusion transformers combining fine-grained dynamic quantization, static-dynamic channel balancing, and metric-decoupled mixed precision, achieving W8A8/W4A8 with near-FP16 quality and 1.4-1.7x latency speedup.

Problem

Post-training quantization of diffusion transformers for text-to-image and text-to-video generation fails under low bit-widths because of large data variation within quantization groups, time-varying channel imbalance, and metric-agnostic sensitivity analysis.

Approach

Combine fine-grained token-wise/channel-wise dynamic activation quantization with a "static-dynamic" channel balancing (SmoothQuant-style scaling for the static timestep-independent part plus Hadamard rotation for the dynamic timestep-dependent part), and allocate mixed precision using metric-decoupled sensitivity (per-layer-type metrics such as VQA, CLIPScore, FlowScore) instead of MSE.

Key results

• ViDiT-Q achieves W8A8 with near-FP16 VBench overall consistency (26.06 vs 26.21 FP16) on OpenSORA, while Q-DiT and PTQ4DiT degrade at W4A8 (e.g., PTQ4DiT overall 11.46) and ViDiT-Q W4A8 reaches 25.94.
• End-to-end CUDA kernel on A100 gives 1.99x peak memory reduction for W8A8 and 2.42x for mixed-precision W4A8, with 1.71x and 1.38x end-to-end latency speedup respectively.
• On PixArt-alpha, ViDiT-Q W4A8 keeps FID/CLIP/ImageReward close to FP16 on COCO, while the static-dynamic channel balancing alone is reported to outperform scaling-only and rotation-only balancing in the OpenSORA ablation (VQA-A 60.2 vs 31.96 and 47.36).

Measured effects reported in the paper

Surprising findings

• Naive MSE-based mixed precision hurts video quality relative to uniform W4A8 in the ablation (VQA-A 53.3 vs 58.3-level results with metric-decoupled allocation), suggesting MSE overemphasizes content changes.
• Dynamic degree can increase under quantization because quantized videos jitter, so higher dynamic degree is not necessarily better in quantized evaluation.

Limitations

• Mixed precision design is still imperfect, and lower activation bit-width is needed to fully exploit 4-bit weight acceleration.
• Current W4A8 kernels upcast 4-bit weights to 8-bit, so memory is saved but computation is not accelerated in the compute-bound regime.
• FVD on small UCF-101 subsets (101 videos) is noisy, motivating the proposed FVD-FP16 metric.

Experimental setup

Datasets: VBench prompt sets OpenSORA prompt set COCO UCF-101

Benchmarks: VBench UCF-101 COCO

Metrics: VBench dimensions CLIPSIM CLIP-Temp DOVER VQA Flow score Temporal flickering FVD FVD-FP16 FID CLIPScore

Baselines: Q-Diffusion Q-DiT PTQ4DiT SmoothQuant QuaRot PTQ4DM AdaRound BRECQ

Builds on: SmoothQuant QuaRot Q-Diffusion PTQ4DiT Q-DiT

Closest papers by meaning

• Q-VDiT: Towards Accurate Quantization and Distillation of Video-Generation Diffusion Transformers (ICML 2025 · similarity 0.95)
• VQ4DiT: Efficient Post-Training Vector Quantization for Diffusion Transformers (AAAI 2025 · similarity 0.93)
• Q-DiT: Accurate Post-Training Quantization for Diffusion Transformers (CVPR 2025 · similarity 0.93)
• PTQ4DiT: Post-training Quantization for Diffusion Transformers (NeurIPS 2024 · similarity 0.92)
• $\text{S}^2$Q-VDiT: Accurate Quantized Video Diffusion Transformer with Salient Data and Sparse Token Distillation (NeurIPS 2025 · similarity 0.92)
• DiRotQ: Rotation-Aware Quantization for 4-bit Diffusion Transformers (NeurIPS 2026 · similarity 0.91)