Research map / Image generation / Fast diffusion sampling
Efficient Personalization of Quantized Diffusion Model without Backpropagation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Personalizes INT8-quantized Stable Diffusion via Textual Inversion using zeroth-order gradient estimates (forward passes only), with subspace gradient projection and partial uniform timestep sampling, cutting training memory to 2.37GB.
Problem
Personalizing diffusion models (Textual Inversion/DreamBooth) requires backpropagation and large activation/gradient memory, which is impractical on memory-constrained edge devices with private data.
Approach
Combines zeroth-order random gradient estimation on an INT8-quantized diffusion model (no dequantization or backprop) with Subspace Gradient, which projects out noisy token dimensions identified from PCA of the token trajectory, and Partial Uniform Timestep Sampling restricted to timesteps where text conditioning matters.
Key results
- ZOODiP reaches 2.37GB training VRAM vs 19.4GB for DreamBooth (up to 8.2x less memory), with CLIP-T 0.287, CLIP-I 0.772, DINO 0.558 comparable to TI (0.285, 0.778, 0.559).
- Versus GF-TI (same memory, 2.37GB), ZOODiP improves CLIP-I from 0.540 to 0.772 and CLIP-T from 0.253 to 0.287.
- ZOODiP trains 2.2x faster than TI for n=1 (20.7 vs 9.42 iter/sec) and 4.2x faster than TuneQDM in training speed comparisons.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace Gradient-Free Textual Inversion with ZOODiP | Stable Diffusion v1.5 personalization · DreamBooth dataset | CLIP-I: 0.540 -> 0.772 | Helped |
| replace Gradient-Free Textual Inversion with ZOODiP | Stable Diffusion v1.5 personalization · DreamBooth dataset | CLIP-T: 0.253 -> 0.287 | Helped |
| use ZOODiP instead of TuneQDM | Stable Diffusion v1.5 personalization · DreamBooth dataset | CLIP-T: 0.289 -> 0.287 | Hurt |
| use ZOODiP instead of TuneQDM | Stable Diffusion v1.5 personalization · DreamBooth dataset | DINO: 0.555 -> 0.558 | Helped |
| replace Textual Inversion with ZOODiP | Stable Diffusion v1.5 personalization · DreamBooth dataset | CLIP-T: 0.285 -> 0.287 | No clear effect |
| add Subspace Gradient (ablation) | ZO optimization for diffusion personalization · DreamBooth subjects <shiny sneaker> and <dog6> | CLIP-I: 0.736 -> 0.747 | Helped |
| add PUTS (ablation) | ZO optimization for diffusion personalization · DreamBooth subjects <shiny sneaker> and <dog6> | CLIP-I: 0.736 -> 0.744 | Helped |
| add Subspace Gradient when PUTS is used (ablation) | ZOODiP · DreamBooth subjects <shiny sneaker> and <dog6> | CLIP-I: 0.744 -> 0.759 | Helped |
| replace PUTS with SNR-based timestep sampling (ablation) | ZO optimization for diffusion personalization · DreamBooth subjects <shiny sneaker> and <dog6> | CLIP-T: 0.266 -> 0.271 | Helped |
| replace SNR-based timestep sampling with PUTS (ablation) | ZO optimization for diffusion personalization · DreamBooth subjects <shiny sneaker> and <dog6> | CLIP-I: 0.719 -> 0.759 | Helped |
| replace uniform timestep sampling with PUTS (ablation) | ZO optimization for diffusion personalization · DreamBooth subjects <shiny sneaker> and <dog6> | CLIP-I: 0.747 -> 0.759 | Helped |
| use PUTS instead of SNR-based sampling (ablation) | ZO optimization for diffusion personalization · DreamBooth subjects <shiny sneaker> and <dog6> | CLIP-T: 0.271 -> 0.266 | Hurt |
Surprising findings
- Sampling timesteps from U(0,500) failed to learn the reference concept, while U(500,1000) succeeded, contrary to uniform sampling assumptions.
- Over two-thirds of trajectory-buffer dimensions can be projected out even with small variance threshold, suggesting token updates lie in a low-dimensional subspace.
Limitations
- Evaluated only on the DreamBooth dataset with Stable Diffusion v1.5 and a single RTX 3090.
- Zeroth-order gradient estimates are noisy, requiring subspace projection to stabilize training.
- Only Textual Inversion-style token personalization; no weight updates of the denoising network.
Experimental setup
Datasets: DreamBooth dataset
Benchmarks: DreamBooth dataset
Metrics: CLIP-T CLIP-I DINO peak GPU memory (GB) training speed (iter/sec) storage (MB)
Baselines: DreamBooth QLoRA PEQA TuneQDM Textual Inversion Gradient-Free Textual Inversion
Builds on: Textual Inversion MeZO QLoRA PEQA TuneQDM
Closest papers by meaning
- PaRa: Personalizing Text-to-Image Diffusion via Parameter Rank Reduction (ICLR 2025 · similarity 0.90)
- EfficientDM: Efficient Quantization-Aware Fine-Tuning of Low-Bit Diffusion Models (ICLR 2024 · similarity 0.89)
- Mining your own secrets: Diffusion Classifier Scores for Continual Personalization of Text-to-Image Diffusion Models (ICLR 2025 · similarity 0.89)
- APT: Adaptive Personalized Training for Diffusion Models with Limited Data (CVPR 2025 · similarity 0.88)
- TCAQ-DM: Timestep-Channel Adaptive Quantization for Diffusion Models (AAAI 2025 · similarity 0.88)
- JeDi: Joint-Image Diffusion Models for Finetuning-Free Personalized Text-to-Image Generation (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.