Research map / Image generation / Text-to-image diffusion
ClassDiffusion: More Aligned Personalization Tuning with Explicit Class Guidance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ClassDiffusion, which adds a semantic preservation loss (cosine distance between CLIP text embeddings of the personalized-token phrase and the class-only phrase) to Custom Diffusion-style fine-tuning to restore compositional ability lost during personalization.
Problem
Tuning-based subject-driven personalization overfits, so customized concepts lose the ability to be composed with other elements (e.g., "a dog wearing a headphone" omits the headphone).
Approach
Attribute the loss of compositionality to semantic drift of the customized token away from its superclass in CLIP text space, and explicitly regularize that drift with a semantic preservation loss that pulls the personalized phrase embedding toward the class phrase embedding.
Key results
- Single-concept CLIP-T 0.300 vs best baseline 0.293 (SVDiff); BLIP2-T 0.460 vs 0.418 (Custom Diffusion/SVDiff).
- Multiple-concept CLIP-T 0.320 and BLIP2-T 0.477 vs Custom Diffusion 0.282 and 0.380.
- DINO-I drops to 0.673 vs DreamBooth 0.700, showing a text-alignment vs subject-fidelity trade-off; user study text win rates above 78% against all baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add semantic preservation loss during personalization tuning | ClassDiffusion · DreamBooth Dataset, single concept | CLIP-T: 0.293 -> 0.300 | Helped |
| Add semantic preservation loss during personalization tuning | ClassDiffusion · DreamBooth Dataset, single concept | CLIP-I: 0.855 -> 0.828 | Hurt |
| Add semantic preservation loss during personalization tuning | ClassDiffusion · DreamBooth Dataset, single concept | DINO-I: 0.700 -> 0.673 | Hurt |
| Add semantic preservation loss during personalization tuning | ClassDiffusion · DreamBooth Dataset, single concept | BLIP2-T: 0.418 -> 0.460 | Helped |
| Add semantic preservation loss during personalization tuning | ClassDiffusion · DreamBooth Dataset, single concept | TIFA: 0.835 -> 0.843 | Helped |
| Add semantic preservation loss during personalization tuning | ClassDiffusion · DreamBooth Dataset, multiple concepts | CLIP-T: 0.282 -> 0.320 | Helped |
| Add semantic preservation loss during personalization tuning | ClassDiffusion · DreamBooth Dataset, multiple concepts | CLIP-I: 0.813 -> 0.821 | Helped |
| Add semantic preservation loss during personalization tuning | ClassDiffusion · DreamBooth Dataset, multiple concepts | DINO-I: 0.636 -> 0.604 | Hurt |
| Add semantic preservation loss during personalization tuning | ClassDiffusion · DreamBooth Dataset, multiple concepts | BLIP2-T: 0.380 -> 0.477 | Helped |
| Increase SPL weight (ablation) | ClassDiffusion · DreamBooth Dataset | CLIP-T: 0.299 -> 0.301 | Helped |
| Increase SPL weight (ablation) | ClassDiffusion · DreamBooth Dataset | DINO-I: 0.677 -> 0.661 | Hurt |
| Use SPL weight 100 instead of lower weights (ablation) | ClassDiffusion · Prompt: a photo of a dog wearing a headphone | Compositional ability | Helped |
Surprising findings
- CLIP-T is reported to hardly reflect actual personalization performance, motivating the BLIP2-T metric.
- The SPL weight is reported as insensitive: CLIP-T barely changes across weights 0.01 to 100 while DINO-I decreases slightly.
Limitations
- Unclear whether the approach transfers to human face personalization.
- Choosing an appropriate class (center) word for composite categories requires experimentation.
- Lower DINO-I than some baselines, indicating a trade-off in concept fidelity.
Experimental setup
Datasets: DreamBooth Dataset Textual Inversion Dataset CustomConcept101
Benchmarks: DreamBooth Dataset TIFA
Metrics: CLIP-T CLIP-I DINO-I BLIP2-T TIFA user study win rate
Baselines: DreamBooth Textual Inversion Custom Diffusion NeTI SVDiff Prospect
Builds on: Custom Diffusion DreamBooth Stable Diffusion AnimateDiff
Closest papers by meaning
- How to Continually Adapt Text-to-Image Diffusion Models for Flexible Customization? (NeurIPS 2024 · similarity 0.88)
- Direct Consistency Optimization for Robust Customization of Text-to-Image Diffusion models (NeurIPS 2024 · similarity 0.87)
- Non-confusing Generation of Customized Concepts in Diffusion Models (ICML 2024 · similarity 0.87)
- ConceptGuard: Continual Personalized Text-to-Image Generation with Forgetting and Confusion Mitigation (CVPR 2025 · similarity 0.87)
- LoRACLR: Contrastive Adaptation for Customization of Diffusion Models (CVPR 2025 · similarity 0.86)
- Attention Calibration for Disentangled Text-to-Image Personalization (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.