Research map / Image generation / Text-to-image diffusion
Personalized Residuals for Concept-Driven Text-to-Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes personalized residuals (LoRA-style low-rank residuals on output projection conv layers of Stable Diffusion, ~1.2M params, no regularization images) plus localized attention-guided sampling that applies the residuals only where cross-attention localizes the concept.
Problem
Concept-driven personalization of text-to-image diffusion models is costly in parameters and training time, often needs regularization images, and struggles with recontextualization such as background changes due to overfitting.
Approach
Learn low-rank residuals only for the output projection conv layers of each transformer block (about 0.1% of Stable Diffusion parameters) without regularization images, and at inference binarize summed cross-attention maps for the concept token and macro class to apply the personalized features only in the concept region while using the original model elsewhere.
Key results
- Personalized residuals reach DINO image alignment 0.5671 with 1.2M parameters, above DreamBooth (0.5212, 983M) and Custom Diffusion (0.4669, 19M).
- Human study: users preferred the method over Textual Inversion for text alignment 81.85% of the time and image alignment 61.96%.
- Trained in about 3 minutes for 150 iterations on a single A100 without regularization images.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply LAG sampling | Personalized Residuals · CustomConcept101 | CLIP text similarity: 0.7193 -> 0.7220 | Helped |
| Apply LAG sampling | Personalized Residuals · CustomConcept101 | DINO image similarity: 0.5671 -> 0.5411 | Hurt |
| Apply residuals to key and value weights instead of output projection weights (ablation) | Personalized Residuals · CustomConcept101 | DINO image similarity: 0.5671 -> 0.5353 | Hurt |
| Apply residuals to input projection convolution weights instead of output projection weights (ablation) | Personalized Residuals · CustomConcept101 | DINO image similarity: 0.5671 -> 0.5333 | Hurt |
Surprising findings
- CLIP text alignment scores did not necessarily correlate with human preference in the AMT study.
- Localized sampling is not always preferred; it was preferred for image alignment but not for text alignment.
Limitations
- Localized sampling is not always the best choice, e.g., for changing the color of a concept.
- Relies on cross-attention maps being high quality, and is sensitive to the choice of macro class.
- Inherits pretrained model biases, such as mixing up attributes in the prompt.
Experimental setup
Datasets: CustomConcept101
Benchmarks: CustomConcept101
Metrics: CLIP text alignment CLIP image alignment DINO image alignment human preference (AMT)
Baselines: Textual Inversion DreamBooth Custom Diffusion ViCo Stable Diffusion
Builds on: LoRA Stable Diffusion DreamBooth Custom Diffusion Textual Inversion
Closest papers by meaning
- Concept Weaver: Enabling Multi-Concept Fusion in Text-to-Image Models (CVPR 2024 · similarity 0.88)
- PaRa: Personalizing Text-to-Image Diffusion via Parameter Rank Reduction (ICLR 2025 · similarity 0.88)
- How to Continually Adapt Text-to-Image Diffusion Models for Flexible Customization? (NeurIPS 2024 · similarity 0.88)
- Unlearning Concepts in Diffusion Model via Concept Domain Correction and Concept Preserving Gradient (AAAI 2025 · similarity 0.88)
- TweedieMix: Improving Multi-Concept Fusion for Diffusion-based Image/Video Generation (ICLR 2025 · similarity 0.87)
- Localized Concept Erasure for Text-to-Image Diffusion Models Using Training-Free Gated Low-Rank Adaptation (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.