Research map / Image generation / Text-to-image diffusion
MS-Diffusion: Multi-subject Zero-shot Image Personalization with Layout Guidance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MS-Diffusion, a frozen-SDXL, layout-guided, zero-shot multi-subject image personalization method using a grounding resampler and masked multi-subject cross-attention, trained on an in-house 3.6M video dataset.
Problem
Zero-shot personalized text-to-image generation with multiple reference subjects suffers from detail loss, subject neglect, subject overcontrol, subject-subject conflicts, and loss of text fidelity.
Approach
A grounding resampler initializes learnable queries with text embeddings of entities and Fourier embeddings of boxes to extract subject features, and a multi-subject cross-attention applies per-subject box masks on image cross-attention (with dummy background tokens) so each subject acts only in its layout region while text cross-attention stays untouched; the base diffusion model is frozen.
Key results
- On DreamBench single-subject, MS-Diffusion reports DINO 0.671 and CLIP-T 0.321 (zero-shot), the highest among zero-shot methods listed, with CLIP-I 0.792 slightly below Kosmos-G (0.822) and IP-Adapter (0.810).
- On MS-Bench multi-subject, MS-Diffusion achieves CLIP-T 0.341 vs 0.316 for λ-ECLIPSE and 0.303 for SSR-Encoder, with DINO 0.425 tied for best and M-DINO 0.108.
- Ablation shows removing multi-subject cross-attention drops multi-subject CLIP-T from 0.341 to 0.309, and removing the resampler drops single-subject DINO from 0.671 to 0.583.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the resampler instead of omitting it (ablation) | MS-Diffusion · DreamBench single-subject | DINO: 0.583 -> 0.671 | Helped |
| add the resampler instead of omitting it (ablation) | MS-Diffusion · MS-Bench multi-subject | DINO: 0.372 -> 0.425 | Helped |
| add the grounding resampler instead of omitting it (ablation) | MS-Diffusion · DreamBench single-subject | DINO: 0.646 -> 0.671 | Helped |
| add the grounding resampler instead of omitting it (ablation) | MS-Diffusion · MS-Bench multi-subject | DINO: 0.389 -> 0.425 | Helped |
| add multi-subject cross-attention instead of omitting it (ablation) | MS-Diffusion · MS-Bench multi-subject | CLIP-T: 0.309 -> 0.341 | Helped |
| use explicit layout guidance rather than omitting it and using IAL (ablation) | MS-Diffusion · MS-Bench multi-subject | CLIP-T: 0.305 -> 0.341 | Helped |
| use explicit layout guidance rather than omitting it and using IAL and TAL (ablation) | MS-Diffusion · MS-Bench multi-subject | CLIP-T: 0.316 -> 0.341 | Helped |
Surprising findings
- Adding an image attention loss for layout guidance (without explicit layout) barely helps personalization, with DINO dropping to 0.577 in single-subject.
- Multi-subject CLIP-I of MS-Diffusion (0.698) is lower than λ-ECLIPSE (0.724) and SSR-Encoder (0.725).
Limitations
- Box-based layout is imprecise and layout must be given explicitly at inference; generated positions are not always accurate.
- CLIP image encoder loses fine details for uncommon or complex subjects; background of subject images leaks into outputs.
- Training data is in-house and not released; MS-Bench is evaluation-only and its release is not stated.
Experimental setup
Datasets: in-house video dataset (3.6M clips) DreamBench MS-Bench
Benchmarks: DreamBench MS-Bench
Metrics: CLIP-I DINO CLIP-T M-DINO
Baselines: SSR-Encoder λ-ECLIPSE IP-Adapter BLIP-Diffusion ELITE Emu2 Kosmos-G Textual Inversion DreamBooth Mix-of-Show Cones2 Break-A-Scene
Builds on: IP-Adapter Stable Diffusion XL GLIGEN Flamingo
Closest papers by meaning
- R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation (ICLR 2024 · similarity 0.89)
- Harmonizing Visual and Textual Embeddings for Zero-Shot Text-to-Image Customization (AAAI 2025 · similarity 0.88)
- Large-Scale Text-to-Image Model with Inpainting is a Zero-Shot Subject-Driven Image Generator (CVPR 2025 · similarity 0.88)
- Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.88)
- JeDi: Joint-Image Diffusion Models for Finetuning-Free Personalized Text-to-Image Generation (CVPR 2024 · similarity 0.87)
- DynASyn: Multi-Subject Personalization Enabling Dynamic Action Synthesis (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.