Research map / Image generation / Text-to-image diffusion
Orthogonal Adaptation for Modular Customization of Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Orthogonal Adaptation, a LoRA variant that freezes a shared random orthogonal basis B and trains only A per concept so independently customized Stable Diffusion concept residuals can be merged by summation with little crosstalk.
Problem
Modular customization of text-to-image diffusion models: merging independently fine-tuned concept models into one model that renders multiple personal concepts without identity loss or heavy merge-time compute.
Approach
Represent each concept's weight residual as a low-rank product A_i B_i^T where B_i is a fixed, approximately orthogonal basis (columns sampled from a shared random orthogonal matrix) and only A_i is trained, so residuals of different concepts interfere minimally when summed (FedAvg-style merging).
Key results
- After merging, the method achieves image alignment 0.741 and identity alignment 0.745 (+0.005 from single to merged), the highest among compared methods, while DB-LoRA (FedAvg) drops to identity alignment 0.098 after merging.
- Merging takes under 1 second versus about 15 minutes for Mix-of-Show gradient fusion, with comparable merged identity quality.
- Single-concept fine-tuning takes about 10-15 minutes on two A6000 GPUs; rank r=20 is used.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use orthogonal adaptation instead of unconstrained FedAvg merging | Stable Diffusion customization · Custom concept dataset, merged | Identity Alignment: .740 → .745 | Helped |
| Use orthogonal adaptation instead of unconstrained FedAvg merging | Stable Diffusion customization · Custom concept dataset, merged | Image Alignment: .748 → .741 | Hurt |
| Use orthogonal adaptation instead of unconstrained FedAvg merging | Stable Diffusion customization · Custom concept dataset, merged | Text Alignment: .624 → .644 | Helped |
| Constrain concept weight bases to be orthogonal (ablation) | Orthogonal Adaptation · Two merged concepts | Identity preservation | Helped |
| Increase the number of orthogonally merged concepts (ablation) | Orthogonal Adaptation · Models with varying numbers of merged concepts | Concept identity | No clear effect |
Surprising findings
- Restricting training to A with a frozen orthogonal B causes little loss in single-concept fidelity, attributed to over-parameterization of diffusion models.
- Randomized Gaussian basis produced more crosstalk than the shared randomized orthogonal basis.
Limitations
- Complex compositions and interactions between multiple custom concepts remain challenging; concepts can be entangled or ignored.
- Existing fine-tuned LoRAs cannot be adapted post hoc to enforce orthogonality; the method requires changing the fine-tuning process.
- Evaluation is limited to a custom dataset of 12 human identities.
Experimental setup
Datasets: custom dataset of 12 concept identities, 16 images each
Benchmarks: custom 12-identity concept dataset
Metrics: image alignment (CLIP image similarity) text alignment (CLIP text-image similarity) identity alignment (ArcFace detection rate)
Baselines: P+ Custom Diffusion DreamBooth-LoRA (FedAvg) Mix-of-Show (Gradient Fusion) Mix-of-Show (FedAvg)
Builds on: LoRA DreamBooth FedAvg Mix-of-Show Custom Diffusion
Closest papers by meaning
- LoRACLR: Contrastive Adaptation for Customization of Diffusion Models (CVPR 2025 · similarity 0.90)
- Direct Consistency Optimization for Robust Customization of Text-to-Image Diffusion models (NeurIPS 2024 · similarity 0.89)
- How to Continually Adapt Text-to-Image Diffusion Models for Flexible Customization? (NeurIPS 2024 · similarity 0.89)
- Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluation (ICLR 2024 · similarity 0.89)
- CURE: Concept Unlearning via Orthogonal Representation Editing in Diffusion Models (NeurIPS 2025 · similarity 0.88)
- APT: Adaptive Personalized Training for Diffusion Models with Limited Data (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.