Research map / Image generation / Text-to-image diffusion
Concept Conductor: Orchestrating Multiple Personalized Concepts in Text-to-Image Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Training-free multi-concept customization framework that runs separate sampling paths for each single-concept custom model and corrects layouts via self-attention guidance from a reference image, fusing concepts with shape-aware masks.
Problem
Generating images with multiple personalized concepts suffers from attribute leakage between concepts and layout confusion (concept omission, subject redundancy, appearance truncation), especially for visually similar concepts.
Approach
Isolate the denoising paths of the base model and each single-concept ED-LoRA model (multipath sampling), align self-attention layouts to a reference image via gradient guidance, and inject each concept's self- and cross-attention features into the base model using shape-aware masks refined by clustering self-attention maps.
Key results
- Concept Conductor achieves CLIP-I 0.9190 and DINO 0.8569 versus 0.8928 and 0.7961 for Mix-of-Show, the strongest baseline.
- ImageReward of 1.2542 versus 0.3944 for Mix-of-Show; omission rate (n<2) 0.0075 versus 0.3400 for Mix-of-Show.
- User study: text alignment 4.74 and image alignment 4.375 out of 5, the highest among compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Concept Conductor instead of Mix-of-Show | Multi-concept customization · 10 visually similar concept pairs | CLIP-I: 0.8928 -> 0.9190 | Helped |
| Use Concept Conductor instead of Mix-of-Show | Multi-concept customization · 10 visually similar concept pairs | CLIP-T: 0.2962 -> 0.3107 | Helped |
| Use Concept Conductor instead of Mix-of-Show | Multi-concept customization · 10 visually similar concept pairs | Omission rate (n<2): 0.3400 -> 0.0075 | Helped |
| Use Concept Conductor instead of Mix-of-Show | Multi-concept customization · 10 visually similar concept pairs | Redundancy rate (n>2): 0.0400 -> 0.0350 | Helped |
| Add layout alignment (ablation) | Concept Conductor · Multi-concept customization (10 visually similar pairs) | CLIP-T: 0.3020 -> 0.3107 | Helped |
| Add self-attention features to concept injection (ablation) | Concept Conductor · Multi-concept customization (10 visually similar pairs) | CLIP-I: 0.8940 -> 0.9190 | Helped |
| Add cross-attention features to concept injection (ablation) | Concept Conductor · Multi-concept customization (10 visually similar pairs) | CLIP-I: 0.9051 -> 0.9190 | Helped |
| Add mask refinement (ablation) | Concept Conductor · Multi-concept customization (10 visually similar pairs) | CLIP-I: 0.9148 -> 0.9190 | Helped |
Surprising findings
- Ablation shows removing layout alignment raises redundancy and omission rates substantially (w/o LA omission 0.0575 vs 0.0075 full model).
Limitations
- Poor quality for small subjects such as small faces due to VAE detail loss.
- Considerable computational overhead from alternating LoRA loading and backpropagation during sampling.
- Requires per-concept ED-LoRA training and a reference layout image.
Experimental setup
Datasets: Mix-of-Show dataset DreamBooth dataset CustomConcept101
Benchmarks: 30-concept multi-concept customization dataset (10 similar pairs, 5 prompts each, 8 seeds)
Metrics: CLIP-T ImageReward CLIP-I DINO SegSim omission/redundancy counting user study
Baselines: Custom Diffusion Cones 2 Mix-of-Show
Builds on: ED-LoRA Mix-of-Show Cones 2 Custom Diffusion local-prompt-mixing
Closest papers by meaning
- MC^2: Multi-concept Guidance for Customized Multi-concept Generation (CVPR 2025 · similarity 0.90)
- Concept Weaver: Enabling Multi-Concept Fusion in Text-to-Image Models (CVPR 2024 · similarity 0.89)
- Attention Calibration for Disentangled Text-to-Image Personalization (CVPR 2024 · similarity 0.87)
- FreeCustom: Tuning-Free Customized Image Generation for Multi-Concept Composition (CVPR 2024 · similarity 0.87)
- LaTexBlend: Scaling Multi-concept Customized Generation with Latent Textual Blending (CVPR 2025 · similarity 0.87)
- ConceptGuard: Continual Personalized Text-to-Image Generation with Forgetting and Confusion Mitigation (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.