Research map / Image generation / Text-to-image diffusion
Zero-Shot Adaptation of Parameter-Efficient Fine-Tuning in Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ProLoRA, a training-free method that transfers pre-trained LoRA/DoRA/FouRA adapters between diffusion models (e.g., SDXL to SSD-1B) by projecting source adapter components onto subspace and null space of matched target weight layers.
Problem
LoRA adapters are tied to their base model, so migrating them to a new or distilled base model normally requires retraining, which is often impractical given compute limits or unavailable training data.
Approach
Decompose the source LoRA update into column/row-subspace and null-space components of the source weights, then project each component onto the matching subspaces of the target weights (selected by subspace similarity), yielding a closed-form transfer without any training data.
Key results
- On style LoRA transfer from SDXL to SSD-1B (Origami), ProLoRA gives HPSv2 0.256 and CSD-MMD 0.0245, matching the trained LoRA (0.244, 0.0245) and far better than Copy w/ SS LoRA (CSD-MMD 0.1663).
- Concept LoRA transfer from SDXL to SSD-1B reaches DINOv2 0.501 versus 0.539 for trained LoRA and 0.475 for copied LoRA.
- Pro-LCM-LoRA on SSD-1B achieves DINOv2 0.944 versus 0.885 for copy LCM-LoRA, with HPSv2 0.315 vs 0.329 for trained LCM-LoRA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| transfer style LoRA from SDXL using ProLoRA instead of directly copying it | SSD-1B on Origami · Origami | CSD-MMD: 0.1663 -> 0.0245 | Helped |
| add nullspace projection to the transfer (ablation) | ProLoRA on SSD-1B · Origami | CSD-MMD: 0.1344 -> 0.0245 | Helped |
| add subspace projection to the transfer (ablation) | ProLoRA on SSD-1B · Origami | CSD-MMD: 0.2120 -> 0.0245 | Helped |
| apply ProLoRA only to modules with non-square weight matrices (ablation) | ProLoRA on SSD-1B · Origami | CSD-MMD: 0.0245 -> 0.1824 | Hurt |
| replace full transfer with direct LoRA copying | SSD-1B concept transfer · DreamBooth concepts | DINOv2: 0.501 -> 0.475 | Hurt |
| transfer LCM-LoRA with ProLoRA instead of copying it | SSD-1B · BlueFire prompts | DINOv2: 0.885 -> 0.944 | Helped |
| reduce LoRA rank from 32 to 1 (ablation) | ProLoRA on SD Eff-v1.0 · BlueFire | CSD-MMD: 0.0025 -> 0.0017 | Helped |
| raise subspace similarity threshold from 0.8 to 1.0 (ablation) | ProLoRA on SD Eff-v1.0 · BlueFire | CSD-MMD: 0.0025 -> 0.0082 | Hurt |
| use chained transfer through RV-3 instead of direct transfer (ablation) | ProLoRA to Eff v1.0 · Origami | CSD-MMD: 0.0025 -> 0.0045 | Hurt |
| initialize DreamBooth fine-tuning with ProLoRA rather than random initialization | SSD-1B concept LoRA at 250 iterations · DreamBooth concepts | DINOv2: 0.368 -> 0.524 | Helped |
| transfer LoRA with ProLoRA rather than train from scratch | TinyLlama 2.5T on E2E-NLG · E2E-NLG | ROUGE-1: 0.7882 -> 0.7881 | No clear effect |
| transfer LoRA with ProLoRA rather than train from scratch | TinyLlama 2.5T on SamSum · SamSum | ROUGE-1: 0.3461 -> 0.3432 | No clear effect |
Surprising findings
- Iterative transfer across a chain of models (SD1.5 to RV3 to Eff v1.0) degrades CSD-MMD on Origami (0.0025 to 0.0045), attributed to error accumulation.
- Transfer at rank 1 collapses on BlueFire (HPSv2 0.126 vs 0.265 for trained LoRA), showing the method is lossy at low rank.
Limitations
- Transfer is lossy and performance degrades at low LoRA ranks and over chained transfers.
- Results are mostly on a few style/concept datasets and mostly on SD/SDXL-family models; a metric (CSD-MMD) is introduced and used primarily by the authors.
- Transfer for cross-architecture or mismatched-dimension layers relies on a linear projection to a common subspace, with limited analysis.
Experimental setup
Datasets: BlueFire Origami Styles Paintings DreamBooth
Benchmarks: BlueFire Paintings Origami DreamBooth E2E-NLG SamSum
Metrics: HPSv2 LPIPS CSD-MMD DINOv2 CLIP-T CLIP-I ROUGE
Baselines: LoRA trained from scratch Copy LoRA LoRA-X X-Adapter ProLoRA w/o NS
Builds on: LoRA LoRA-X DoRA FouRA LCM-LoRA
Closest papers by meaning
- SaRA: High-Efficient Diffusion Model Fine-tuning with Progressive Sparse Low-Rank Adaptation (ICLR 2025 · similarity 0.89)
- Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluation (ICLR 2024 · similarity 0.88)
- LoRA-X: Bridging Foundation Models with Training-Free Cross-Model Adaptation (ICLR 2025 · similarity 0.88)
- IntLoRA: Integral Low-rank Adaptation of Quantized Diffusion Models (ICML 2025 · similarity 0.88)
- Exploring Low-Dimensional Subspace in Diffusion Models for Controllable Image Editing (NeurIPS 2024 · similarity 0.88)
- Enabling Preference-driven Unlearning in Few-step Distilled Text-to-Image Diffusion Models (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.