Research map / Architectures and efficiency / Pruning and efficient tuning
IntLoRA: Integral Low-rank Adaptation of Quantized Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IntLoRA, which fine-tunes quantized diffusion models with integer-type low-rank parameters so merged weights stay quantized and need no post-training quantization at inference.
Problem
Existing LoRA-on-quantized-weight recipes merge into FP16 weights, so an extra post-training quantization step is needed at deployment, which causes large performance drops at low bit-width.
Approach
Adaptation-quantization separation (zero-initialized LoRA plus a gradient-free auxiliary matrix R), multiplicative low-rank adaptation that decouples pre-trained and adaptation quantizers, and variance matching control to make the adaptation term log2-quantizable, yielding two variants: MUL (integer multiplication) and SHIFT (bit-shifting).
Key results
- On subject-driven generation at W4A8, IntLoRA-MUL reaches DINO 0.4242 vs 0.2136 for QLoRA and 0.3722 for IR-QLoRA.
- On 4-bit S2I controllable generation, IntLoRA-MUL achieves FID 30.87, outperforming IR-QLoRA (35.83) and QLoRA (71.75).
- On GSM8K with Llama3-8B at 8-bit, IntLoRA-MUL reaches 64.23% accuracy vs 64.06% for QLoRA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use integer-arithmetic MUL instead of PTQ-based QLoRA | DreamBooth subject-driven generation W8A8 · DreamBooth subject-driven generation W8A8 | DINO: 0.4153 -> 0.4498 | Helped |
| Use integer-arithmetic MUL instead of PTQ-based QLoRA | DreamBooth subject-driven generation W4A8 · DreamBooth subject-driven generation W4A8 | CLIP-I: 0.6134 -> 0.6913 | Helped |
| Use MUL rather than EfficientDM | LDM-4 W4A4 diffusion quantization · ImageNet 256×256 | FID: 13.42 -> 10.43 | Helped |
| Replace Gaussian auxiliary-matrix sampling with Laplace sampling (ablation) | IntLoRA subject-driven generation · DreamBooth subject-driven generation | DINO: 0.4135 -> 0.4492 | Helped |
| Replace Cauchy auxiliary-matrix sampling with Laplace sampling (ablation) | IntLoRA subject-driven generation · DreamBooth subject-driven generation | DINO: 0.1367 -> 0.4492 | Helped |
| Replace StudentT auxiliary-matrix sampling with Laplace sampling (ablation) | IntLoRA subject-driven generation · DreamBooth subject-driven generation | DINO: 0.2935 -> 0.4492 | Helped |
| Increase the VMC smoothing factor alpha from near zero toward 1.5 (ablation) | MUL · Diffusion model adaptation | performance | Helped |
| Use Laplace rather than heavier-tailed auxiliary-matrix distributions (ablation) | IntLoRA · DreamBooth subject-driven generation | CLIP-I: 0.6756 -> 0.6980 | Helped |
| Increase LoRA rank (ablation) | IntLoRA · Diffusion adaptation tasks | performance | Helped |
| Set auxiliary variance too large or too small instead of using VMC (ablation) | SHIFT · Diffusion model adaptation | convergence | Hurt |
Surprising findings
- Naive QLoRA collapses at W4A8 on subject-driven generation (DINO 0.2136), while the SHIFT log2 variant still works at W4A8.
- On ImageNet W4A4 diffusion quantization, IntLoRA-SHIFT (FID 20.20) underperforms EfficientDM (13.42), while IntLoRA-MUL (10.43) outperforms it.
Limitations
- Quantized low-rank weights remain in FP16 during tuning, so training is not integer-only.
- The SHIFT variant's hyperparameter alpha and auxiliary-matrix distribution must be tuned by search.
- Evaluated mainly on Stable Diffusion-scale models with small rank and subsets of datasets; the SHIFT variant lags MUL on ImageNet W4A4.
Experimental setup
Datasets: DreamBooth ADE20K CelebA-HQ COCO StyleDrop MetaMathQA ImageNet GSM8K
Benchmarks: DreamBooth subject-driven generation ADE20K S2I CelebA-HQ L2F COCO C2I StyleDrop style generation GSM8K ImageNet 256x256 generation
Metrics: DINO CLIP-I CLIP-T LPIPS FID IS sFID precision accuracy
Baselines: LoRA QLoRA QA-LoRA IR-QLoRA EfficientDM
Builds on: LoRA QLoRA EfficientDM Dreambooth
Closest papers by meaning
- EfficientDM: Efficient Quantization-Aware Fine-Tuning of Low-Bit Diffusion Models (ICLR 2024 · similarity 0.90)
- Pioneering 4-Bit FP Quantization for Diffusion Models: Mixup-Sign Quantization and Timestep-Aware Fine-Tuning (CVPR 2025 · similarity 0.89)
- Zero-Shot Adaptation of Parameter-Efficient Fine-Tuning in Diffusion Models (ICML 2025 · similarity 0.88)
- Qua2SeDiMo: Quantifiable Quantization Sensitivity of Diffusion Models (AAAI 2025 · similarity 0.88)
- TCAQ-DM: Timestep-Channel Adaptive Quantization for Diffusion Models (AAAI 2025 · similarity 0.88)
- MPQ-DM: Mixed Precision Quantization for Extremely Low Bit Diffusion Models (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.