Research map / Architectures and efficiency / Pruning and efficient tuning
SaRA: High-Efficient Diffusion Model Fine-tuning with Progressive Sparse Low-Rank Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SaRA, a PEFT method for diffusion models that fine-tunes the smallest-magnitude (temporarily ineffective) weights as a sparse matrix, with a nuclear-norm low-rank loss, progressive parameter readjustment, and unstructural backpropagation to cut memory.
Problem
Adapting large pre-trained diffusion models (Stable Diffusion series) to downstream tasks effectively and efficiently, while avoiding overfitting, layer-specific design, added inference latency, and high memory cost of selective fine-tuning.
Approach
Observes that the smallest-magnitude weights contribute little to generation and that their ineffectiveness stems from training randomness, so it re-uses them by fine-tuning only those sparse weights, constrains their rank with a nuclear-norm loss, reselects remaining below-threshold weights progressively, and backpropagates only through selected entries.
Key results
- On SD 1.5 with 50M trainable params, SaRA achieves mean FID 134.96 and mean VLHI 1.55, versus 150.74 FID and VLHI 1.45 for LoRA.
- SaRA reports the best mean VLHI in most SD 1.5/2.0/3.0 settings, e.g., mean VLHI 1.69 at 20M on SD 1.5 vs 1.52 for LoRA.
- The paper claims more than 40% GPU memory savings relative to LoRA and selective PEFT methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add progressive parameter adjustment (ablation) | SaRA without PPA and rank loss · Expedition | FID: 134.75 -> 130.95 | Helped |
| add progressive parameter adjustment (ablation) | SaRA without PPA and rank loss · Expedition | CLIP Score: 27.01 -> 26.66 | Hurt |
| add progressive parameter adjustment (ablation) | SaRA without PPA and rank loss · Expedition | VLHI: 1.16 -> 1.56 | Helped |
| add nuclear-norm rank loss (ablation) | SaRA with PPA · Expedition | FID: 130.95 -> 131.56 | Hurt |
| add nuclear-norm rank loss (ablation) | SaRA with PPA · Expedition | CLIP Score: 26.66 -> 27.34 | Helped |
| add nuclear-norm rank loss (ablation) | SaRA with PPA · Expedition | VLHI: 1.56 -> 1.79 | Helped |
| add progressive parameter adjustment (ablation) | SaRA with rank loss · Expedition | VLHI: 0.89 -> 1.79 | Helped |
| tune the largest absolute-value parameters rather than the smallest (ablation) | SaRA · Expedition | CLIP Score: 27.34 -> 25.42 | Hurt |
| tune randomly selected parameters rather than the smallest absolute-value parameters (ablation) | SaRA · Expedition | VLHI: 1.79 -> 0.97 | Hurt |
| reselect remaining below-threshold parameters in the later training stage (ablation) | SaRA · Fine-tuning | remaining ineffective parameters: 15% -> 2% | Helped |
Surprising findings
- Zeroing the smallest 10-20% of weights barely hurts generation, and in some cases slightly improves FID over the unmodified model.
- Fine-tuning the largest-magnitude parameters gives a good FID but the worst CLIP score, indicating severe prior disruption.
Limitations
- Results depend on hyperparameters such as threshold and learning rate schedule, which require tuning across SD versions.
- Evaluation is limited to small style datasets and a few subject-customization cases, with FID and CLIP proxies.
- Performance on SD 3.0 favors more learnable parameters, suggesting sensitivity to model scale.
Experimental setup
Datasets: ImageNet FFHQ CelebA-HQ CIVITAI style datasets (Barbie, Cyberpunk, ElementFire, Expedition, Hornify) DreamBooth subjects (Dog, Clock, Backpack)
Benchmarks: FID on CIVITAI style datasets CLIP score VLHI T2I-CompBench++ B-VQA DreamBooth image customization CLIP-I/CLIP-T
Metrics: FID CLIP score VLHI B-VQA CLIP-I
Baselines: LoRA AdaptFormer LT-SFT Full-finetune DoRA DiffPruning Textual Inversion DreamBooth
Builds on: LoRA AdaptFormer LT-SFT DiffPruning PaFi
Closest papers by meaning
- Zero-Shot Adaptation of Parameter-Efficient Fine-Tuning in Diffusion Models (ICML 2025 · similarity 0.89)
- ALTER: All-in-One Layer Pruning and Temporal Expert Routing for Efficient Diffusion Generation (NeurIPS 2025 · similarity 0.89)
- Attention-Driven Training-Free Efficiency Enhancement of Diffusion Models (CVPR 2024 · similarity 0.89)
- Analyzing and Improving the Training Dynamics of Diffusion Models (CVPR 2024 · similarity 0.89)
- Neural Diffusion Models (ICML 2024 · similarity 0.89)
- Diffusion Models Without Attention (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.