Research map / Image generation / Text-to-image diffusion
Attention Calibration for Disentangled Text-to-Image Personalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DisenDiff, which learns multiple disentangled concepts from one image in Stable Diffusion by adding attention-map constraints (modifier-class binding, separate-and-strengthen, suppression) on cross-attention while updating only WK/WV and new token embeddings.
Problem
Personalizing text-to-image diffusion models to capture multiple novel concepts from a single reference image, where existing methods produce cluttered cross-attention maps, losing visual fidelity and suffering cross-concept interference.
Approach
Calibrates cross-attention maps with an IoU-based modifier-class binding loss, an IoU-based separate-and-strengthen loss between class tokens, and an element-wise self-multiplication suppression plus Gaussian smoothing, so each concept is bound to its class and decoupled from others.
Key results
- Averaged across ten datasets, DisenDiff achieves the highest image-alignment score, with the largest gap on Concept2, while keeping text-alignment comparable to other methods.
- Textual Inversion has the highest text-alignment but the lowest image-alignment; Custom Diffusion is the closest competitor, notably weaker on Concept2.
- Ablations show removing the Lbind or Ls&s loss significantly reduces image alignment for both concepts; removing suppression harms Concept2 image alignment.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply attention calibration with modifier-class binding, separate-and-strengthen loss, and suppression | DisenDiff · Ten-dataset evaluation, image alignment | image alignment | Helped |
| Apply attention calibration with modifier-class binding, separate-and-strengthen loss, and suppression | DisenDiff · Ten-dataset evaluation, text alignment | text alignment | No clear effect |
| Use DisenDiff rather than Custom Diffusion | DisenDiff · Independent Concept2 | image alignment | Helped |
| Remove the separate-and-strengthen loss (ablation) | DisenDiff · Independent Concept1 and Concept2 | image alignment | Hurt |
| Remove modifier-class binding loss (ablation) | DisenDiff · Independent Concept1 and Concept2 | image alignment | Hurt |
| Remove the suppression strategy (ablation) | DisenDiff · Independent Concept2 | image alignment | Hurt |
| Remove the suppression strategy (ablation) | DisenDiff · Independent Concept1 and Concept2 | text alignment | Hurt |
| Remove the Gaussian filter (ablation) | DisenDiff · Combined concepts | image alignment | Hurt |
| Apply suppression twice instead of once (ablation) | DisenDiff · Concepts | image alignment | Hurt |
| Apply constraints to the average attention maps across all scales instead of the 16 × 16 scale (ablation) | DisenDiff · Independent Concept2 | image alignment | Hurt |
| Update WQ in addition to WK and WV (ablation) | DisenDiff · Concepts | image alignment | Helped |
| Update WQ in addition to WK and WV (ablation) | DisenDiff · Concepts | text alignment | Hurt |
Surprising findings
- Constraining attention averaged over all scales hurt reconstruction of Concept2, and updating WQ, WK, WV improved appearance retention but significantly reduced text alignment.
Limitations
- Struggles to disentangle fine-grained categories when two subjects of the same category co-exist in one image (e.g., two dog breeds).
- Performance degrades considerably with three concepts from a single image.
- Relies on existing T2I model limitations for complex multi-concept scenarios.
Experimental setup
Datasets: ten custom single-image datasets (people, animals, furniture, people with pets/toys) LAION-5B (regularization images)
Benchmarks: custom ten-dataset single-image multi-concept benchmark
Metrics: CLIP image alignment CLIP text alignment
Baselines: Textual Inversion DreamBooth Custom Diffusion
Builds on: Custom Diffusion Stable Diffusion Prompt-to-Prompt DreamBooth
Closest papers by meaning
- MC^2: Multi-concept Guidance for Customized Multi-concept Generation (CVPR 2025 · similarity 0.89)
- Concept Conductor: Orchestrating Multiple Personalized Concepts in Text-to-Image Synthesis (AAAI 2025 · similarity 0.87)
- ConceptBed: Evaluating Concept Learning Abilities of Text-to-Image Diffusion Models (AAAI 2024 · similarity 0.87)
- Concept Weaver: Enabling Multi-Concept Fusion in Text-to-Image Models (CVPR 2024 · similarity 0.87)
- DisEnvisioner: Disentangled and Enriched Visual Prompt for Customized Image Generation (ICLR 2025 · similarity 0.87)
- DreamMatcher: Appearance Matching Self-Attention for Semantically-Consistent Text-to-Image Personalization (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.