Research map / Image generation / Text-to-image diffusion
Selectively Informative Description can Reduce Undesired Embedding Entanglements in Text-to-Image Personalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SID, a GPT-4-generated train description that adds informative specifications of undesired objects (not the subject) to optimization-based text-to-image personalization, reducing undesired embedding entanglement.
Problem
Text-to-image personalization from few reference images overfits to biases (background, nearby objects, tied objects, substance, pose) in references, entangling non-subject information into the subject identifier embedding and hurting text alignment.
Approach
Instead of describing only the subject's class in training captions, add informative descriptions of undesired objects in the reference images (generated by multimodal GPT-4) so that non-subject content aligns with its own text and does not attach to the identifier token [v]; the subject itself is deliberately not described in detail.
Key results
- SID-integrated optimization-based models form the Pareto boundary in subject-alignment, non-subject-disentanglement, and text-alignment metrics for multiple reference images (Fig. 8).
- Human evaluation with 130 participants (5,200 responses) is consistent with the metric analysis.
- Cross-attention maps show SID-integrated DreamBooth focuses on the subject while DreamBooth's identifier attends to undesired objects.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SID to DreamBooth, Custom Diffusion, and SVDiff | Optimization-based personalization models · Multiple reference images | non-subject-disentanglement | Helped |
| Add SID to DreamBooth, Custom Diffusion, and SVDiff | Optimization-based personalization models · Multiple reference images | text-alignment | Helped |
| Add SID to DreamBooth, Custom Diffusion, and SVDiff | Optimization-based personalization models · Multiple reference images | subject-alignment | Helped |
| Add SID to optimization-based personalization models | Optimization-based personalization models · Single reference image | non-subject-disentanglement | Helped |
| Add SID to optimization-based personalization models | Optimization-based personalization models · Single reference image | text-alignment | Helped |
| Add informative specifications of undesired objects to the train description (ablation) | Personalization with Case 1 baseline descriptions · Examples in Fig. 3 | embedding entanglement reduction | Helped |
| Add class identifications of undesired objects to the train description (ablation) | Personalization with Case 1 baseline descriptions · Examples in Fig. 3 | embedding entanglement reduction | Helped |
| Add informative specifications of the subject to the train description (ablation) | Personalization with SID (Case 3) · Examples in Fig. 3 | subject identity preservation | Hurt |
| Use GPT-4-generated SID instead of BLIP-2-generated captions (ablation) | SID generation · VLM comparison | subject-alignment | Helped |
| Use GPT-4-generated SID instead of BLIP-2-generated captions (ablation) | SID generation · VLM comparison | text-alignment | Helped |
| Use SID instead of a negative prompt at inference | DreamBooth · Examples in Fig. 10 | non-subject-disentanglement | Helped |
| Use SID instead of a negative prompt at inference | DreamBooth · Examples in Fig. 10 | text-alignment | Helped |
Surprising findings
- The widely used image-alignment score is significantly influenced by background entanglement, so it is inappropriate for measuring undesired entanglement.
- Including subject specifications in the train description (Case 4) disentangles subject details from the identifier, hurting subject preservation.
Limitations
- Performance depends on VLM caption quality; GPT-4 occasionally fails to follow instructions.
- Strong facial expressions of the subject can remain entangled unless the SID includes expression details.
- Only integrated with optimization-based models; not evaluated with encoder-based models.
Experimental setup
Datasets: DreamBooth dataset Custom Diffusion dataset images from three websites
Benchmarks: DreamBooth dataset Custom Diffusion dataset
Metrics: subject-alignment non-subject-disentanglement text-alignment cross-attention maps human evaluation
Baselines: DreamBooth Custom Diffusion SVDiff Textual Inversion ELITE BLIP-Diffusion negative prompt segmentation mask
Builds on: DreamBooth Custom Diffusion SVDiff Textual Inversion
Closest papers by meaning
- AttnDreamBooth: Towards Text-Aligned Personalized Text-to-Image Generation (NeurIPS 2024 · similarity 0.87)
- Decoupled Textual Embeddings for Customized Image Generation (AAAI 2024 · similarity 0.86)
- CoRe: Context-Regularized Text Embedding Learning for Text-to-Image Personalization (AAAI 2025 · similarity 0.86)
- Harmonizing Visual and Textual Embeddings for Zero-Shot Text-to-Image Customization (AAAI 2025 · similarity 0.86)
- Attention Calibration for Disentangled Text-to-Image Personalization (CVPR 2024 · similarity 0.86)
- DisenBooth: Identity-Preserving Disentangled Tuning for Subject-Driven Text-to-Image Generation (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.