Research map / Image generation / Text-to-image diffusion
Identity Decoupling for Multi-Subject Personalization of Text-to-Image Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MuDI, which uses SAM-segmented subjects for Seg-Mix data augmentation during DreamBooth-style SDXL personalization and mean-shifted noise initialization, plus a Detect-and-Compare metric, to avoid identity mixing across multiple subjects.
Problem
Multi-subject text-to-image personalization suffers from identity mixing, where generated subjects acquire combined attributes, especially when subjects are visually similar.
Approach
Use SAM- and OWLv2-segmented subjects both as a data augmentation (Seg-Mix: randomly composing segmented subjects on a blank background, allowing overlap) during DreamBooth-style LoRA training and as a mean-shifted initial latent at inference, with descriptive class nouns and a new Detect-and-Compare metric for multi-subject fidelity.
Key results
- On the new benchmark, MuDI reaches D&C-DS 0.637 versus 0.432 for Cut-Mix and 0.371 for DreamBooth.
- In human evaluation, MuDI achieves about twice the success rate of Cut-Mix at avoiding identity mixing and is preferred over 70% against the strongest baseline.
- MuDI achieves the highest ImageReward (0.770) and CLIP score (0.263) among compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MuDI instead of Cut-Mix | SDXL multi-subject personalization · Multi-subject personalization | D&C-DS: 0.432 -> 0.637 | Helped |
| use MuDI instead of DreamBooth | SDXL multi-subject personalization · Multi-subject personalization | ImageReward†: 0.579 -> 0.770 | Helped |
| use MuDI instead of Cut-Mix | SDXL multi-subject personalization · Multi-subject personalization | CLIPs†: 0.225 -> 0.263 | Helped |
| add Seg-Mix training augmentation (ablation) | MuDI · Multi-subject personalization | D&C-DS: 0.475 -> 0.637 | Helped |
| add Seg-Mix training augmentation (ablation) | MuDI · Multi-subject personalization | D&C-DINO: 0.481 -> 0.610 | Helped |
| add inference initialization (ablation) | MuDI · Multi-subject personalization | D&C-DS: 0.477 -> 0.637 | Helped |
| add inference initialization (ablation) | MuDI · Multi-subject personalization | D&C-DINO: 0.480 -> 0.610 | Helped |
| use descriptive classes (ablation) | MuDI · Multi-subject personalization | D&C-DS: 0.556 -> 0.637 | Helped |
| use descriptive classes (ablation) | MuDI · Multi-subject personalization | D&C-DINO: 0.558 -> 0.610 | Helped |
| allow subject overlap during Seg-Mix (ablation) | MuDI · Multi-subject personalization | identity separation | Helped |
| apply iterative training using selected high-quality samples | MuDI · Corgi and Chow Chow | D&C-DS: 0.613 to 0.672 | Helped |
| use LLM-guided rather than random initialization layouts | MuDI · Complex subject interactions | interaction alignment | Helped |
Surprising findings
- Cut-Mix with a negative prompt reduces artifacts but causes over-saturation, so the authors did not use negative prompts for Cut-Mix evaluation.
- Layout conditioning via region control is ineffective on SDXL, and single-subject metrics like DreamSim give high scores to mixed-identity images.
Limitations
- Success rate decreases as the number of personalized subjects increases, especially for highly similar subjects.
- Evaluation relies on a small dataset of 8 subject combinations and 5 prompts each.
Experimental setup
Datasets: DreamBench CustomConcept101
Benchmarks: Multi-subject personalization evaluation dataset (8 combinations, 400 images per method)
Metrics: D&C-DS D&C-DINO ImageReward CLIP score human success rate
Baselines: DreamBooth DreamBooth with region control Cut-Mix Textual Inversion Custom Diffusion
Builds on: DreamBooth Cut-Mix Segment Anything LoRA
Closest papers by meaning
- JeDi: Joint-Image Diffusion Models for Finetuning-Free Personalized Text-to-Image Generation (CVPR 2024 · similarity 0.87)
- DynASyn: Multi-Subject Personalization Enabling Dynamic Action Synthesis (AAAI 2025 · similarity 0.86)
- FaceA-Net: Facial Attribute-Driven ID Preserving Image Generation Network (AAAI 2025 · similarity 0.85)
- HyperDreamBooth: HyperNetworks for Fast Personalization of Text-to-Image Models (CVPR 2024 · similarity 0.85)
- MS-Diffusion: Multi-subject Zero-shot Image Personalization with Layout Guidance (ICLR 2025 · similarity 0.85)
- RectifID: Personalizing Rectified Flow with Anchored Classifier Guidance (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.