Research map / Image generation / Text-to-image diffusion
Decoupled Textual Embeddings for Customized Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DETEX, which represents each training image with an image-shared subject embedding plus image-specific pose and background embeddings (produced by CLIP-fed attribute mappers), trained jointly with a cross-attention loss and random background filtering to disentangle the concept for customized text-to-image generation.
Problem
Customized text-to-image generation from a few images tends to overfit and entangle subject-unrelated information (background, pose) with the learned concept, limiting editability and composition into new scenes.
Approach
Represent each image with multiple word embeddings: one shared subject embedding plus image-specific pose and background embeddings produced by attribute mappers from CLIP image features, supervised by a cross-attention loss tied to subject masks and a joint training scheme that randomly removes backgrounds. At inference only the subject embedding is used, optionally with the pose/background embed
Key results
- On DreamBench, DETEX achieves the highest CLIP-T (0.3249) and foreground-only CLIP-I (0.7688) and DINO-I (0.6121), versus Custom Diffusion's 0.3143 CLIP-T.
- User study with 20 users per view shows DETEX preferred over Custom Diffusion, DreamBooth, ViCo and SVDiff on text alignment (54.95-71.45%) and image alignment (54.15-79.55%).
- Ablation shows removing the attribute mapper, attention loss, or joint training lowers foreground subject fidelity (DINO-I (FG) 0.5789/0.5766/0.5625 vs 0.5797 for full model).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add attribute mappers for pose and background embeddings (ablation) | DETEX · DreamBench | CLIP-I (FG): 0.7563 -> 0.7695 | Helped |
| Add attribute mappers for pose and background embeddings (ablation) | DETEX · DreamBench | DINO-I (FG): 0.5789 -> 0.5797 | Helped |
| Add attribute mappers for pose and background embeddings (ablation) | DETEX · DreamBench | CLIP-T: 0.3295 -> 0.3301 | Hurt |
| Add cross-attention loss (ablation) | DETEX · DreamBench | CLIP-T: 0.3287 -> 0.3301 | Helped |
| Add cross-attention loss (ablation) | DETEX · DreamBench | CLIP-I (FG): 0.7548 -> 0.7695 | Helped |
| Add cross-attention loss (ablation) | DETEX · DreamBench | DINO-I (FG): 0.5766 -> 0.5797 | Helped |
| Add joint training strategy (ablation) | DETEX · DreamBench | CLIP-I (FG): 0.7403 -> 0.7695 | Helped |
| Add joint training strategy (ablation) | DETEX · DreamBench | DINO-I (FG): 0.5625 -> 0.5797 | Helped |
| Add joint training strategy (ablation) | DETEX · DreamBench | CLIP-T: 0.3327 -> 0.3301 | Hurt |
| Add masked diffusion loss (ablation) | DETEX · DreamBench | CLIP-T: 0.3301 -> 0.3289 | Hurt |
| Add masked diffusion loss (ablation) | DETEX · DreamBench | CLIP-I (FG): 0.7695 -> 0.7498 | Hurt |
| Add masked diffusion loss (ablation) | DETEX · DreamBench | DINO-I (FG): 0.5797 -> 0.5624 | Hurt |
Surprising findings
- Full-image CLIP-I and DINO-I favor DreamBooth and Custom Diffusion, but these gaps reverse on foreground-only metrics, which the authors argue reflects background entanglement.
- Completely discarding backgrounds during training (gamma=1.0) causes the model to overfit to blank backgrounds and produce solid-color backgrounds.
Limitations
- One-shot learning struggles to separate shared and image-specific concepts, causing entanglement between [V], [P], and [B].
- Masked diffusion loss and full background filtration hurt pose and background quality, so the method relies on careful loss/probability tuning (gamma=0.5).
- Subject masks require SAM plus manual selection of target masks.
Experimental setup
Datasets: DreamBench regularization images generated by Stable Diffusion
Benchmarks: DreamBench
Metrics: CLIP-T CLIP-I CLIP-I (FG) DINO-I DINO-I (FG) user preference
Baselines: Textual Inversion DreamBooth Custom Diffusion SVDiff ViCo DreamBooth+LoRA
Builds on: Custom Diffusion Stable Diffusion Textual Inversion DisenBooth
Closest papers by meaning
- Salient Concept-Aware Generative Data Augmentation (NeurIPS 2025 · similarity 0.88)
- An Image is Worth Multiple Words: Discovering Object Level Concepts using Multi-Concept Prompt Learning (ICML 2024 · similarity 0.88)
- Harmonizing Visual and Textual Embeddings for Zero-Shot Text-to-Image Customization (AAAI 2025 · similarity 0.88)
- DisenBooth: Identity-Preserving Disentangled Tuning for Subject-Driven Text-to-Image Generation (ICLR 2024 · similarity 0.87)
- Continuous, Subject-Specific Attribute Control in T2I Models by Identifying Semantic Directions (CVPR 2025 · similarity 0.87)
- Selectively Informative Description can Reduce Undesired Embedding Entanglements in Text-to-Image Personalization (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.