Research map / Vision-language and multimodal / CLIP and zero-shot
Non-confusing Generation of Customized Concepts in Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CLIF, which fine-tunes the CLIP text encoder contrastively on over-segmented, augmented customized-concept data before fine-tuning the diffusion U-Net with a shared LoRA, to reduce inter-concept confusion in multi-concept customization.
Problem
Inter-concept visual confusion (identity loss, attribute leakage, concept missing) when composing multiple user-provided customized concepts in text-guided diffusion models, especially with few-shot concept examples.
Approach
Revisits the two TGDM stages and argues prior customization methods only fine-tune the decoder; CLIF instead contrastively fine-tunes the CLIP text encoder on augmented image-text pairs (global, region, and mixed augmentations built from SAM segmentation, generated contexts, and GPT-4 captions) so concept token embeddings are decoupled, then fine-tunes the U-Net with a shared LoRA.
Key results
- CLIF reaches multi-concept text alignment 0.599 and image alignment 0.761, outperforming Mix-of-Show (0.526 / 0.713) and other baselines in the multi-concept setting.
- Adding CLIF to Custom Diffusion raises its multi-concept image alignment from 0.715 to 0.730 and text alignment from 0.511 to 0.535.
- Removing global augmentation lowers multi-concept image alignment from 0.761 to 0.729.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply CLIF instead of Mix-of-Show | Multi-concept generation · Multi-concept | text alignment: 0.526 -> 0.599 | Helped |
| Apply CLIF instead of Mix-of-Show | Multi-concept generation · Multi-concept | image alignment: 0.713 -> 0.761 | Helped |
| Add CLIF to TI | TI · Multi-concept | text alignment: 0.507 -> 0.528 | Helped |
| Add CLIF to Custom | Custom · Multi-concept | text alignment: 0.511 -> 0.535 | Helped |
| Add CLIF to Custom | Custom · Single-concept | image alignment: 0.749 -> 0.774 | Helped |
| Add G-Aug (ablation) | CLIF · Multi-concept | image alignment: 0.729 -> 0.761 | Helped |
| Add R-Aug (ablation) | CLIF · Multi-concept | image alignment: 0.753 -> 0.761 | Helped |
| Add R-Aug (ablation) | CLIF · Single-concept | image alignment: 0.774 -> 0.787 | Helped |
| Add M-Aug (ablation) | CLIF · Multi-concept | image alignment: 0.740 -> 0.761 | Helped |
| Add M-Aug (ablation) | CLIF · Multi-concept | text alignment: 0.579 -> 0.599 | Helped |
Surprising findings
- Single-concept text alignment of CLIF (0.630) is only comparable to Mix-of-Show (0.629), so gains are concentrated in multi-concept settings.
Limitations
- Experiments focus on two-concept compositions; scaling to more concepts is said to require extending the dataset and has limitations discussed only in the appendix.
- Relies on augmentation pipelines (SAM segmentation, GPT-4 captions, generated contexts) whose quality affects training.
- Evaluation uses CLIP-based alignment metrics and a self-curated character set.
Experimental setup
Datasets: 18 curated customized characters (real-world, 3D-animated, 2D-animated) images generated by pre-trained diffusion model for general contexts
Benchmarks: custom 18-character multi-concept set DreamBench
Metrics: text alignment (CLIP) image alignment (CLIP) attendance
Baselines: Text-Inversion Custom Diffusion DreamBooth Mix-of-Show PACGen
Builds on: CLIP Stable Diffusion Text Inversion Custom Diffusion DreamBooth Mix-of-Show
Closest papers by meaning
- The Hidden Language of Diffusion Models (ICLR 2024 · similarity 0.90)
- Concept Weaver: Enabling Multi-Concept Fusion in Text-to-Image Models (CVPR 2024 · similarity 0.89)
- LLM4GEN: Leveraging Semantic Representation of LLMs for Text-to-Image Generation (AAAI 2025 · similarity 0.88)
- Progressive Compositionality in Text-to-Image Generative Models (ICLR 2025 · similarity 0.88)
- MC^2: Multi-concept Guidance for Customized Multi-concept Generation (CVPR 2025 · similarity 0.88)
- Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.