Research map / Image generation / Text-to-image diffusion
Multi-Concept Customization of Text-to-Image Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune cross-attention key and value matrices rather than all U-Net parameters | Custom Diffusion · Single-concept (averaged across datasets) | Image-alignment: 0.748 -> 0.775 | Helped |
| Fine-tune cross-attention key and value matrices rather than all U-Net parameters | Custom Diffusion · Single-concept (averaged across datasets) | Text-alignment: 0.795 -> 0.795 | No clear effect |
| Jointly train both concepts with cross-attention key and value updates | Custom Diffusion · Multi-concept (five composition pairs) | Text-alignment: 0.783 -> 0.801 | Helped |
| Jointly train both concepts with cross-attention key and value updates | Custom Diffusion · Multi-concept (five composition pairs) | Image-alignment: 0.695 -> 0.706 | Helped |
| Add image resizing and size-related prompt augmentation during training (ablation) | Custom Diffusion · Single-concept (averaged across datasets) | Image-alignment: 0.736 -> 0.775 | Helped |
| Add the real-image regularization dataset (ablation) | Custom Diffusion · Single-concept (averaged across datasets) | KID (validation): 32.64 -> 20.96 | Helped |
| Replace real-image regularization with generated images (ablation) | Custom Diffusion · Single-concept (averaged across datasets) | KID (validation): 20.96 -> 34.70 | Hurt |
| Do not optimize the modifier token after rare-token initialization (ablation) | Custom Diffusion · Single-concept (averaged across datasets) | Image-alignment: 0.774 -> 0.733 | Hurt |
| Initialize the modifier token from a normal distribution instead of a rare token (ablation) | Custom Diffusion · Single-concept (averaged across datasets) | Image-alignment: 0.774 -> 0.765 | Hurt |
| Initialize the modifier token from a normal distribution instead of a rare token (ablation) | Custom Diffusion · Single-concept (averaged across datasets) | Text-alignment: 0.795 -> 0.805 | Helped |
| Compress the parameter updates using only the top 60% of singular values (ablation) | Custom Diffusion · Single-concept customization | Image-alignment | No clear effect |
| Compress the parameter updates using only the top 1% of singular values (ablation) | Custom Diffusion · Single-concept customization | Image-alignment | Hurt |
Closest papers by meaning
- How to Continually Adapt Text-to-Image Diffusion Models for Flexible Customization? (NeurIPS 2024 · similarity 0.93)
- Concept Weaver: Enabling Multi-Concept Fusion in Text-to-Image Models (CVPR 2024 · similarity 0.92)
- Non-confusing Generation of Customized Concepts in Diffusion Models (ICML 2024 · similarity 0.90)
- Generating Images of Rare Concepts Using Pre-trained Diffusion Models (AAAI 2024 · similarity 0.90)
- DreamDistribution: Learning Prompt Distribution for Diverse In-distribution Generation (ICLR 2025 · similarity 0.90)
- Orthogonal Adaptation for Modular Customization of Diffusion Models (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.