Research map / Image generation / Text-to-image diffusion
Mining your own secrets: Diffusion Classifier Scores for Continual Personalization of Text-to-Image Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes continual personalization of Stable Diffusion by regularizing LoRA training with diffusion classifier (DC) scores, via EWC Fisher estimates (EWC DC) and a double-distillation function-space method (DSC), outperforming C-LoRA on replay-free sequences.
Problem
Personalizing text-to-image diffusion models on sequentially arriving concepts without access to earlier data causes forgetting of previous concepts, and existing methods like C-LoRA balance new-concept acquisition and retention poorly.
Approach
Uses diffusion classifier scores (class-conditional denoising likelihoods) to improve EWC's Fisher information estimate (parameter-space) and to drive a double-teacher distillation (DSC, adapted from deep model consolidation) that regularizes function space, with no replay data and zero parameter overhead over C-LoRA.
Key results
- On Custom Concept after 6 tasks, DSC EWC DC achieves the best KID (140.18e-5), A_MMD (94.1e-3) and BwT_MMD (-92.44) versus C-LoRA's 173.8, 117.2 and -107.47.
- On Landmarks after 10 tasks, DSC EWC DC gets KID 51.22e-5 and A_MMD 36.09e-3 versus C-LoRA's 104.63 and 68.77, and BwT_MMD -3.85 versus -16.47.
- Method adds zero parameter overhead over C-LoRA in trainable (0.09%) and stored (100.54%) parameters, and is replay-free.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add DC-score regularization to EWC (ablation) | LoRA EWC · Custom Concept (6 tasks) | BwT_MMD: -99.34 -> -102.53 | No clear effect |
| Add DSC function-space consolidation to EWC DC (ablation) | LoRA continual personalization · Custom Concept (6 tasks) | KID: 154.25 -> 140.18 | Helped |
| Add DSC function-space consolidation to EWC DC (ablation) | LoRA continual personalization · Landmarks (10 tasks) | KID: 56.98 -> 51.22 | Helped |
| Add DSC function-space consolidation to EWC DC (ablation) | LoRA continual personalization · Textual Inversion household objects (9 tasks) | KID: 139.66 -> 121.08 | Helped |
| Add DSC function-space consolidation to EWC DC (ablation) | LoRA continual personalization · Celeb-A (10 tasks) | KID: 127.0 -> 119.5 | Helped |
| Add DC scores to EWC (ablation) | LoRA EWC · Landmarks (10 tasks) | A_MMD: 44.53 -> 38.43 | Helped |
| Add DC scores to EWC (ablation) | LoRA EWC · Textual Inversion household objects (9 tasks) | A_MMD: 88.21 -> 82.75 | Helped |
| Add DC scores to EWC (ablation) | LoRA EWC · Celeb-A (10 tasks) | A_MMD: 95.07 -> 87.06 | Helped |
| Use DSC without EWC (ablation) | LoRA continual personalization · Custom Concept (6 tasks) | KID: 156.91 -> 187.2 | Hurt |
| Use DSC without EWC (ablation) | LoRA continual personalization · Landmarks (10 tasks) | KID: 66.76 -> 111.69 | Hurt |
| Use DSC without the noise score-matching loss (ablation) | DSC consolidation · Custom Concept (task 2) | generation quality | Hurt |
| Use k=2 concepts, excluding the common prior concept (ablation) | EWC DC · Custom Concept | performance | Hurt |
Surprising findings
- Naive DSC without previous task data undergoes huge forgetting (BwT_MMD -105.79 on Custom Concept), and DSC consolidated with DC scores alone yields unintelligible images.
- The relative forgetting metric F_MMD can be misleading: a weak learner that barely learns new tasks shows low forgetting, motivating the absolute BwT_MMD metric.
Limitations
- DC-score regularization increases training cost: about 5.3 s per iteration for EWC and 5.7 s for DSC versus 0.8 s for C-LoRA on 6 tasks.
- Results cover four concept datasets with 3 seeds, with hyperparameters tuned on the six-task Custom Concept sequence.
- DSC is more sensitive to the number of consolidation iterations than EWC DC.
Experimental setup
Datasets: CustomConcept101 Google Landmarks v2 Textual Inversion household objects dataset CelebA
Benchmarks: CustomConcept101 Google Landmarks v2 Textual Inversion household objects CelebA
Metrics: KID A_MMD BwT_MMD CLIP I2I F_MMD parameters trained parameters stored
Baselines: Textual Inversion Custom Diffusion C-LoRA LoRA sequential LoRA merge EWC CD EWC
Builds on: C-LoRA Custom Diffusion Elastic Weight Consolidation Deep Model Consolidation Diffusion classifier
Closest papers by meaning
- APT: Adaptive Personalized Training for Diffusion Models with Limited Data (CVPR 2025 · similarity 0.91)
- How to Continually Adapt Text-to-Image Diffusion Models for Flexible Customization? (NeurIPS 2024 · similarity 0.89)
- PaRa: Personalizing Text-to-Image Diffusion via Parameter Rank Reduction (ICLR 2025 · similarity 0.89)
- Where's Waldo: Diffusion Features For Personalized Segmentation and Retrieval (NeurIPS 2024 · similarity 0.89)
- Efficient Personalization of Quantized Diffusion Model without Backpropagation (CVPR 2025 · similarity 0.89)
- On improved Conditioning Mechanisms and Pre-training Strategies for Diffusion Models (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.