Research map / Image generation / Text-to-image diffusion
Concept Weaver: Enabling Multi-Concept Fusion in Text-to-Image Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Concept Weaver composes multiple customized text-to-image diffusion concepts at inference time by generating a template image and fusing per-concept personalized models into it via masked cross-attention and feature injection.
Problem
Generating images that combine multiple personalized concepts in text-to-image diffusion models remains challenging, with concept blending and missing concepts as concepts scale beyond two or three.
Approach
A tuning-free cascade: a template image (from SDXL or a real photo) is inverted, its self-attention and residual features are injected, and per-concept Custom Diffusion models are fused in masked regions using concept-aware prompts, feature-space mixing, and concept-free suppression, without any joint training or weight merging.
Key results
- Concept Weaver reaches CLIP text-sim 0.3804 and image-sim 0.8124, outperforming Mix-of-show (0.3634 / 0.7984) and Custom Diffusion (0.3595 / 0.7875) on 15 concepts in 5 combinations.
- In a 20-participant user study, Concept Weaver scores 4.70 on text match and 4.64 on concept match on a 5-point scale, versus 3.44 and 3.39 for Mix-of-show.
- Full pipeline takes about 60 seconds per image on a single RTX 3090 with SD2.1 at 768x768 and 50 sampling steps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Concept Weaver instead of Mix-of-show | Multi-concept image generation · Multi-concept generation | Text sim: 0.3634 -> 0.3804 | Helped |
| use Concept Weaver instead of Mix-of-show | Multi-concept image generation · Multi-concept generation | Image sim: 0.7984 -> 0.8124 | Helped |
| use Concept Weaver instead of Mix-of-show | Multi-concept image generation · Human preference study | Text match: 3.44 -> 4.70 | Helped |
| use Concept Weaver instead of Mix-of-show | Multi-concept image generation · Human preference study | Concept match: 3.39 -> 4.64 | Helped |
| use Concept Weaver instead of Mix-of-show | Multi-concept image generation · Human preference study | Realism: 3.78 -> 4.43 | Helped |
| add structural feature injection (ablation) | Concept Weaver · Multi-concept generation | Text sim: 0.3489 -> 0.3804 | Helped |
| add structural feature injection (ablation) | Concept Weaver · Multi-concept generation | Image sim: 0.7739 -> 0.8124 | Helped |
| add concept-free suppression (ablation) | Concept Weaver · Multi-concept generation | Text sim: 0.3727 -> 0.3804 | Helped |
| add concept-free suppression (ablation) | Concept Weaver · Multi-concept generation | Image sim: 0.7936 -> 0.8124 | Helped |
| use masked guidance together with the other Concept Weaver components instead of only mask guidance (ablation) | Concept Weaver · Multi-concept generation | Text sim: 0.3140 -> 0.3804 | Helped |
| use masked guidance together with the other Concept Weaver components instead of only mask guidance (ablation) | Concept Weaver · Multi-concept generation | Image sim: 0.7544 -> 0.8124 | Helped |
| fuse cross-attention features rather than epsilon score outputs (ablation) | Concept Weaver · Multi-concept generation | Text sim: 0.3677 -> 0.3804 | Helped |
Surprising findings
- Realism score of Concept Weaver (4.43) is slightly below Mix-of-show (3.78)? No: Concept Weaver realism is 4.43, above Mix-of-show's 3.78, but Perfusion scored highest on realism among baselines (2.70) lower than Concept Weaver.
- Naively mixing per-concept score estimates (epsilon mix) produces boundary artifacts, and dense segmentation masks deform outputs, so dilated masks were needed.
Limitations
- Requires a template image and per-concept fine-tuning before composition, adding multi-step pipeline cost (about 60 seconds per image).
- Evaluated on a limited set of 15 concepts in 5 combinations, with CLIP-based metrics and a 20-person user study.
- Mask quality from grounding and SAM pipelines governs concept placement; dense masks cause deformation.
Experimental setup
Datasets: Custom Concept101 Blender Open Movie character images
Benchmarks: Custom Concept101
Metrics: CLIP text similarity CLIP image similarity user study (text match, concept match, realism)
Baselines: Textual Inversion Custom Diffusion Perfusion Mix-of-show
Builds on: Custom Diffusion Plug-and-Play diffusion features Segment Anything Model Mix-of-show classifier-free guidance
Closest papers by meaning
- MC^2: Multi-concept Guidance for Customized Multi-concept Generation (CVPR 2025 · similarity 0.91)
- Concept Conductor: Orchestrating Multiple Personalized Concepts in Text-to-Image Synthesis (AAAI 2025 · similarity 0.89)
- How to Continually Adapt Text-to-Image Diffusion Models for Flexible Customization? (NeurIPS 2024 · similarity 0.89)
- LoRACLR: Contrastive Adaptation for Customization of Diffusion Models (CVPR 2025 · similarity 0.89)
- MultiBooth: Towards Generating All Your Concepts in an Image from Text (AAAI 2025 · similarity 0.89)
- Non-confusing Generation of Customized Concepts in Diffusion Models (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.