Research map / Image generation / Text-to-image diffusion
AttnDreamBooth: Towards Text-Aligned Personalized Text-to-Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AttnDreamBooth, a three-stage personalization method (embedding alignment, cross-attention refinement, U-Net identity fine-tuning) with a cross-attention map regularization, improving identity and text alignment over TI+DB, NeTI, ViCo and OFT on Stable Diffusion 2.1.
Problem
Text-to-image personalization trades off identity preservation against text alignment: Textual Inversion overfits the concept and DreamBooth overlooks it in novel prompts, attributed to incorrect embedding alignment of the learned concept token.
Approach
Decouple personalization into three sequential stages, learning the textual embedding for alignment with very few steps, then fine-tuning cross-attention layers for the attention map, then fine-tuning the full U-Net for identity, with the text encoder frozen throughout, plus a regularizer matching mean and variance of [V] and super-category cross-attention maps.
Key results
- Ours achieves identity 0.7257 and text 0.2532, versus TI+DB 0.7017/0.2578, NeTI 0.6901/0.2522, ViCo 0.7507/0.2106, OFT 0.7257/0.2445.
- In a user study with 700 responses from 35 participants, users preferred AttnDreamBooth over TI+DB 68.0%, NeTI 79.4%, ViCo 83.4%, OFT 72.6%.
- Training takes about 20 minutes per concept on a single A100 GPU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use AttnDreamBooth instead of ViCo | Personalized text-to-image generation · 22 concepts, 24 prompts | Identity: 0.7507 -> 0.7257 | Hurt |
| use AttnDreamBooth instead of ViCo | Personalized text-to-image generation · 22 concepts, 24 prompts | Text: 0.2106 -> 0.2532 | Helped |
| use AttnDreamBooth instead of TI+DB | Personalized text-to-image generation · 22 concepts, 24 prompts | Identity: 0.7017 -> 0.7257 | Helped |
| use AttnDreamBooth instead of TI+DB | Personalized text-to-image generation · 22 concepts, 24 prompts | Text: 0.2578 -> 0.2532 | Hurt |
| use AttnDreamBooth instead of OFT | Personalized text-to-image generation · 22 concepts, 24 prompts | Identity: 0.7257 -> 0.7257 | No clear effect |
| use AttnDreamBooth instead of OFT | Personalized text-to-image generation · 22 concepts, 24 prompts | Text: 0.2445 -> 0.2532 | Helped |
| use AttnDreamBooth instead of NeTI | Personalized text-to-image generation · 22 concepts, 24 prompts | Identity: 0.6901 -> 0.7257 | Helped |
| use AttnDreamBooth instead of NeTI | Personalized text-to-image generation · 22 concepts, 24 prompts | Text: 0.2522 -> 0.2532 | Helped |
| add textual embedding optimization (Stage 1) (ablation) | AttnDreamBooth · Personalized text-to-image generation | Embedding alignment | Helped |
| add cross-attention layer fine-tuning (Stage 2) (ablation) | AttnDreamBooth · Personalized text-to-image generation | Text alignment | Helped |
| add full U-Net fine-tuning (Stage 3) (ablation) | AttnDreamBooth · Personalized text-to-image generation | Identity preservation | Helped |
| add cross-attention map regularization (ablation) | AttnDreamBooth · Personalized text-to-image generation | Text alignment | Helped |
Surprising findings
- Textual embedding of TI+DB remains nearly unchanged from initialization (cosine similarity about 0.9997), so joint tuning under-learns embedding alignment.
Limitations
- Fixed training steps across concepts; per-concept tuning could improve results.
- Three-stage training takes about 20 minutes per concept because the full U-Net is fine-tuned for 500 steps.
Experimental setup
Datasets: Textual Inversion concept dataset DreamBooth concept dataset
Benchmarks: TI and DB concept datasets with 24 text prompts
Metrics: CLIP image similarity (identity) CLIP text-image similarity (text alignment) user preference rate
Baselines: TI+DB NeTI ViCo OFT
Builds on: Textual Inversion DreamBooth Custom Diffusion Stable Diffusion
Closest papers by meaning
- InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning (CVPR 2024 · similarity 0.90)
- CoRe: Context-Regularized Text Embedding Learning for Text-to-Image Personalization (AAAI 2025 · similarity 0.90)
- Tailored Visions: Enhancing Text-to-Image Generation with Personalized Prompt Rewriting (CVPR 2024 · similarity 0.88)
- Text Embedding is Not All You Need: Attention Control for Text-to-Image Semantic Alignment with Text Self-Attention Maps (CVPR 2025 · similarity 0.88)
- Discriminative Probing and Tuning for Text-to-Image Generation (CVPR 2024 · similarity 0.88)
- Yo'Chameleon: Personalized Vision and Language Generation (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.