Research map / Image generation / Text-to-image diffusion
DisenBooth: Identity-Preserving Disentangled Tuning for Subject-Driven Text-to-Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
DisenBooth finetunes Stable Diffusion 2-1 with disentangled identity-preserved (textual) and identity-irrelevant (visual, adapter+learnable mask on CLIP image features) embeddings, using weak denoising and contrastive embedding losses for subject-driven generation.
Problem
Subject-driven text-to-image generation suffers because subject identity and identity-irrelevant information (background, pose) are entangled in the learned text embedding, causing overfitting to background and identity drift.
Approach
Denoise each subject image with the sum of a shared textual identity-preserved embedding and an image-specific visual identity-irrelevant embedding (from CLIP image features passed through a learnable feature mask and adapter), and add a weak denoising loss (identity embedding alone, small weight) and a cosine contrastive loss to disentangle the two embeddings; LoRA is used for efficient U-Net tun
Key results
- On DreamBench, DisenBooth reaches CLIP-T 0.330 vs 0.319 for DreamBooth, with DINO 0.675 vs DreamBooth's 0.685.
- Human identity-preservation average rank of 1.37 for DisenBooth vs 2.20 for DreamBooth (lower is better).
- With only 1 training image, DisenBooth gets DINO 0.617 and CLIP-T 0.321, outperforming DreamBooth (0.610 / 0.297).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DisenBooth instead of DreamBooth | subject-driven text-to-image generation · DreamBench | CLIP-T Score: 0.319 -> 0.330 | Helped |
| use DisenBooth instead of DreamBooth | subject-driven text-to-image generation · DreamBench | DINO Score: 0.685 -> 0.675 | Hurt |
| use DisenBooth instead of DreamBooth | subject-driven text-to-image generation · DreamBench | Identity Avg. Rank: 2.20 -> 1.37 | Helped |
| use DisenBooth instead of DreamBooth | subject-driven text-to-image generation · DreamBench | Overall Avg. Rank: 2.45 -> 1.59 | Helped |
| add weak denoising objective (L2) (ablation) | DisenBooth · DreamBench, half split | DINO: 0.684 -> 0.693 | Helped |
| add weak denoising objective (L2) (ablation) | DisenBooth · DreamBench, half split | CLIP-T: 0.329 -> 0.331 | Helped |
| add contrastive embedding objective (L3) (ablation) | DisenBooth · DreamBench, half split | DINO: 0.689 -> 0.693 | Helped |
| add contrastive embedding objective (L3) (ablation) | DisenBooth · DreamBench, half split | CLIP-T: 0.328 -> 0.331 | Helped |
| add learnable feature-level mask (ablation) | DisenBooth · DreamBench, half split | DINO: 0.673 -> 0.693 | Helped |
| add learnable feature-level mask (ablation) | DisenBooth · DreamBench, half split | CLIP-T: 0.330 -> 0.331 | Helped |
| use feature-level mask instead of pixel-level mask (ablation) | DisenBooth · DreamBench, 10 subjects | CLIP-T: 0.327 -> 0.336 | Helped |
| finetune the CLIP text encoder (ablation) | DreamBooth · DreamBench, 20 subjects | CLIP-T: 0.322 -> 0.286 | Hurt |
Surprising findings
- Higher DINO score (identity similarity) can result from overfitting to identity-irrelevant background, so DreamBooth's top DINO score is less meaningful than its low text fidelity suggests.
- Feature-level learnable mask outperformed an explicit pixel-level (foreground) mask, which only separates background and misses pose.
Limitations
- Inherits limitations of the pretrained Stable Diffusion model.
- Only disentangles subject identity from identity-irrelevant information without extra supervision; finer control over pose, background, or style is not addressed.
- Multi-subject customization shows some texture detail loss; non-centered small subjects are hard to preserve.
Experimental setup
Datasets: DreamBench
Benchmarks: DreamBench
Metrics: DINO score CLIP-T score human preference average rank
Baselines: DreamBooth Textual Inversion (TI) InstructPix2Pix pretrained Stable Diffusion Custom Diffusion SVDiff Break-A-Scene ELITE E4T
Builds on: DreamBooth Textual Inversion LoRA Stable Diffusion
Closest papers by meaning
- Decoupled Textual Embeddings for Customized Image Generation (AAAI 2024 · similarity 0.87)
- Dis²Booth: Learning Image Distribution with Disentangled Features for Text-to-Image Diffusion Models (AAAI 2025 · similarity 0.87)
- InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning (CVPR 2024 · similarity 0.87)
- DisEnvisioner: Disentangled and Enriched Visual Prompt for Customized Image Generation (ICLR 2025 · similarity 0.87)
- Transferring disentangled representations: bridging the gap between synthetic and real images (NeurIPS 2024 · similarity 0.86)
- OneActor: Consistent Subject Generation via Cluster-Conditioned Guidance (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.