Research map / Image generation / Text-to-image diffusion
PaRa: Personalizing Text-to-Image Diffusion via Parameter Rank Reduction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PaRa, a parameter-efficient T2I personalization method that reduces the rank of diffusion layer outputs via a QR-derived orthonormal basis, restricting the generation space to balance concept alignment and text editability.
Problem
Personalizing large pretrained text-to-image diffusion models with few target images trades off alignment to the new concept against preserving text editability and generation diversity.
Approach
Instead of adding low-rank updates (LoRA) or scaling singular values, PaRa learns a matrix B, takes its QR-derived orthonormal basis Q, and subtracts the projection QQ^T W0 from the pretrained weights, explicitly shrinking the column space of each layer and thus the generation space; combined PaRa models can be sequentially added and paired with LoRA.
Key results
- PaRa achieves 2x fewer learnable parameters than LoRA and much better image alignment (1-LPIPS) across Dreambooth subjects at comparable text alignment.
- In a human survey with 209 participants, PaRa was preferred over LoRA for single-subject generation at about 91-97% on five subjects (e.g., 96.17% vs 3.83% on bear plushie).
- PaRa at rank 4-8 yields higher average SSIM than vanilla SDXL, indicating reduced generation diversity, and gives more stable single-image editing.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply PaRa rank reduction instead of using the unmodified SDXL model | SDXL 1.0 · DreamBooth single-subject generation, “A [V] SCULPTURE MADE OF GOLD” | average SSIM: 0.185±0.015 -> 0.203±0.008 | Helped |
| Increase the PaRa rank from r = 4 to r = 8 (ablation) | PaRa · DreamBooth single-subject generation, WOLF_PLUSHIE, “A [V] SCULPTURE MADE OF GOLD” | average SSIM: 0.170±0.012 -> 0.160±0.017 | Hurt |
| Increase the PaRa rank from r = 4 to r = 8 (ablation) | PaRa · DreamBooth single-subject generation, RED_CARTOON, “A [V] SCULPTURE MADE OF GOLD” | average SSIM: 0.327±0.014 -> 0.386±0.015 | Helped |
| Increase the PaRa rank from r = 4 to r = 8 (ablation) | PaRa · DreamBooth single-subject generation, GREY_SLOTH_PLUSHIE, “A [V] SCULPTURE MADE OF GOLD” | average SSIM: 0.382±0.013 -> 0.379±0.011 | Hurt |
| Increase the PaRa rank from r = 4 to r = 8 (ablation) | PaRa · DreamBooth single-subject generation, DUCKTOY, “A [V] SCULPTURE MADE OF GOLD” | average SSIM: 0.516±0.010 -> 0.519±0.015 | Helped |
| Use PaRa rather than SVDiff for one-shot single-image editing without DDIM inversion | SDXL 1.0 · Single-image editing | average SSIM | Helped |
| Apply PaRa before LoRA rather than applying PaRa after LoRA (ablation) | Combined PaRa and LoRA · Text-to-image generation with combined personalized weights | generation quality | Helped |
| Increase rank reduction r (ablation) | PaRa · Image editing | reconstruction fidelity | Helped |
| Increase rank reduction r (ablation) | PaRa · Image editing | editability | Hurt |
| Set rank boundary γ around 1/20 to 1/40 rather than using an unsuitable boundary (ablation) | PaRa · Single-subject generation | generation quality | Helped |
| Use a larger PaRa rank for the primary subject than for the secondary subject (ablation) | Combined PaRa models · Multi-subject generation | subject generation quality | Mixed |
| Update different UNet parameter subsets with PaRa instead of using ExCA (ablation) | PaRa · Single-subject generation | target image alignment | Mixed |
Surprising findings
- Applying PaRa before LoRA in combination works better than LoRA first, since LoRA-first variations get eliminated by the subsequent rank reduction.
Limitations
- Only reduces the original output space; cannot expand it when customization requires new directions.
- Rank must be bounded per layer (gamma factor) to avoid model collapse at large r.
- Relatively lower text alignment at high ranks.
Experimental setup
Datasets: DreamBooth dataset
Benchmarks: DreamBooth dataset
Metrics: average SSIM CLIP score 1-LPIPS image alignment human preference survey
Baselines: DreamBooth Textual Inversion SVDiff LoRA Vanilla SDXL
Builds on: LoRA DreamBooth SVDiff Textual Inversion
Closest papers by meaning
- DreamSteerer: Enhancing Source Image Conditioned Editability using Personalized Diffusion Models (NeurIPS 2024 · similarity 0.90)
- Direct Consistency Optimization for Robust Customization of Text-to-Image Diffusion models (NeurIPS 2024 · similarity 0.90)
- Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluation (ICLR 2024 · similarity 0.90)
- Efficient Personalization of Quantized Diffusion Model without Backpropagation (CVPR 2025 · similarity 0.90)
- JeDi: Joint-Image Diffusion Models for Finetuning-Free Personalized Text-to-Image Generation (CVPR 2024 · similarity 0.90)
- Energy-Guided Optimization for Personalized Image Editing with Pretrained Text-to-Image Diffusion Models (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.