Research map / Image generation / Text-to-image diffusion
LoRACLR: Contrastive Adaptation for Customization of Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
LoRACLR merges multiple independently trained LoRA (and ED-LoRA) concept models into one Stable Diffusion model by learning a weight delta with a contrastive objective, without retraining the individual LoRAs.
Problem
Combining multiple personalized LoRA models for multi-concept text-to-image generation causes attribute entanglement and identity loss, and existing methods need retraining, special LoRA formats, segmentation, or original training data.
Approach
Learn an additive weight delta on the base model so that outputs for each concept's LoRA (positive pairs) are attracted to the merged model's outputs while outputs of other concepts' LoRAs (negative pairs) are repelled, using a contrastive loss with an L2 delta regularizer.
Key results
- LoRACLR reaches identity alignment 0.828 after merging (vs 0.745 for Orthogonal Adaptation, 0.706 for MoS FedAvg) on the 12-concept setup in Table 1.
- Image alignment after merging is 0.776 for LoRACLR versus 0.741 for Orthogonal Adaptation.
- Merging 12 LoRAs takes about 5 minutes on one A100, versus about 120 minutes total for Orthogonal Adaptation including per-LoRA fine-tuning; user study identity score 3.42 vs 2.41 for Orthogonal Adaptation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LoRACLR contrastive merging instead of the compared merge methods | Stable Diffusion with personalized LoRAs · 12-concept merge | Image Alignment: + .010 | Helped |
| Use LoRACLR contrastive merging instead of the compared merge methods | Stable Diffusion with personalized LoRAs · 12-concept merge | Identity Alignment: +.029 | Helped |
| Use LoRACLR contrastive merging instead of the compared merge methods | Stable Diffusion with personalized LoRAs · 12-concept merge | Text Alignment: -.003 | Hurt |
| Add LoRACLR contrastive merging rather than Custom Diffusion merging | Stable Diffusion with personalized LoRAs · 12-concept merge | Identity Alignment: -.096 | Helped |
| Use LoRACLR contrastive merging rather than DB-LoRA FedAvg | Stable Diffusion with personalized LoRAs · 12-concept merge | Identity Alignment: -.585 | Helped |
| Use LoRACLR contrastive merging rather than MoS gradient fusion | Stable Diffusion with personalized LoRAs · 12-concept merge | Identity Alignment: -.011 | Helped |
| Use LoRACLR contrastive merging rather than Orthogonal Adaptation | Stable Diffusion with personalized LoRAs · 12-concept merge | Identity Alignment: +.005 | Helped |
| Set the contrastive margin around 0.25–0.5 rather than higher values (ablation) | LoRACLR · Multi-concept compositions | Identity preservation and visual coherence | Helped |
| Increase the contrastive margin above the reported effective range (ablation) | LoRACLR · Multi-concept compositions | Identity preservation and prompt coherence | Hurt |
| Set λdelta to 0.001 rather than other tested values (ablation) | LoRACLR · Multi-concept compositions | Identity preservation and visual coherence | Helped |
| Increase λdelta above 0.001 (ablation) | LoRACLR · Multi-concept compositions | Identity preservation and prompt coherence | Hurt |
| Increase the number of merged concepts across 2, 5, 8, and 12 concepts (ablation) | LoRACLR · Multi-concept compositions | Identity preservation and visual coherence | No clear effect |
Surprising findings
- Text alignment after merging for LoRACLR slightly decreases (.668 to .665) while identity and image alignment increase, and DB-LoRA identity alignment collapses from .683 to .098 after merging.
Limitations
- Performance is bounded by the quality of the underlying individual LoRA models.
- Potential for misuse such as deepfakes.
- Evaluation limited to a single 12-concept identity dataset and Stable Diffusion with ChilloutMix.
Experimental setup
Datasets: 12-concept dataset from Orthogonal Adaptation (16 images per concept)
Benchmarks: 12-concept customization dataset from Orthogonal Adaptation
Metrics: text alignment (CLIP) image alignment (CLIP) identity alignment (ArcFace) user study identity score
Baselines: P+ Custom Diffusion DB-LoRA Mix-of-Show Orthogonal Adaptation
Builds on: LoRA Mix-of-Show Orthogonal Adaptation DreamBooth Stable Diffusion
Closest papers by meaning
- Orthogonal Adaptation for Modular Customization of Diffusion Models (CVPR 2024 · similarity 0.90)
- LoRAShop: Training-Free Multi-Concept Image Generation and Editing with Rectified Flow Transformers (NeurIPS 2025 · similarity 0.90)
- Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluation (ICLR 2024 · similarity 0.89)
- Concept Weaver: Enabling Multi-Concept Fusion in Text-to-Image Models (CVPR 2024 · similarity 0.89)
- How to Continually Adapt Text-to-Image Diffusion Models for Flexible Customization? (NeurIPS 2024 · similarity 0.89)
- Direct Consistency Optimization for Robust Customization of Text-to-Image Diffusion models (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.