Research map / Continual learning / Online and replay-based
How to Continually Adapt Text-to-Image Diffusion Models for Flexible Customization?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CIDM, a concept-incremental text-to-image diffusion model that uses a concept consolidation loss, elastic weight aggregation, and context-controllable synthesis to learn new personalized concepts over time without replay.
Problem
Custom diffusion models assume fixed personalized concepts and suffer catastrophic forgetting and concept neglect when learning a stream of new concepts without storing old training data.
Approach
Learns layer-wise concept tokens and LoRA weights with an orthogonal subspace regularizer plus a shared subspace term (task-specific and task-shared knowledge), merges stored task LoRAs at inference using semantic-relation weights, and adds regional cross-attention with region noise estimation for layout-controlled multi-concept generation.
Key results
- Single-concept IA average of 78.0 on SD-1.5 and 79.5 on SDXL, versus 76.9 and 77.8 for the best baseline CLoRA (stated tables).
- TA average of 74.8 on SD-1.5 and 80.0 on SDXL, outperforming baselines by 1.2%-4.8% TA per the paper's claim.
- Memory for storing task learners is about 0.25% of the pretrained model parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CIDM instead of CLoRA | SD-1.5 · Single-concept customization (V1–V10) | IA: 76.9 -> 78.0 | Helped |
| Use CIDM instead of CLoRA | SDXL · Single-concept customization (V1–V10) | IA: 77.8 -> 79.5 | Helped |
| Use CIDM instead of CLoRA | SD-1.5 · Single-concept customization (V1–V10) | TA: 73.6 -> 74.8 | Helped |
| Use CIDM instead of L2DM | SDXL · Single-concept customization (V1–V10) | TA: 78.7 -> 80.0 | Helped |
| Add EWA (ablation) | Baseline · Single-concept customization (V1–V10), SD-1.5 | IA: 74.6 -> 77.3 | Helped |
| Add TSP (ablation) | CIDM with EWA and TSH · Single-concept customization (V1–V10), SD-1.5 | IA: 77.4 -> 77.9 | Helped |
| Add TSH (ablation) | CIDM with EWA and TSP · Single-concept customization (V1–V10), SD-1.5 | IA: 77.7 -> 77.9 | Helped |
| Add the context-controllable synthesis strategy (RCA and RNE) (ablation) | CIDM · Multi-concept customization | generation performance | Helped |
Limitations
- Struggles to continually learn large numbers of semantically similar concepts, as acknowledged in the limitations section.
- Evaluation uses a small custom dataset of 10 tasks with only 3-5 image-text pairs per task.
Experimental setup
Datasets: CIL dataset (10 concept-incremental customization tasks, constructed by authors) CLoRA/Custom Diffusion concept images (V1-V5, V7, V9 object concepts)
Benchmarks: CIL dataset
Metrics: image-alignment (CLIP image similarity) text-alignment (CLIP text-image similarity)
Baselines: Finetuning EWC LWF RPY CLoRA L2DM LoRA-M LoRA-C
Builds on: LoRA Custom Diffusion Mix-of-Show EWC Learning without Forgetting
Closest papers by meaning
- ConceptGuard: Continual Personalized Text-to-Image Generation with Forgetting and Confusion Mitigation (CVPR 2025 · similarity 0.92)
- Concept Weaver: Enabling Multi-Concept Fusion in Text-to-Image Models (CVPR 2024 · similarity 0.89)
- Mining your own secrets: Diffusion Classifier Scores for Continual Personalization of Text-to-Image Diffusion Models (ICLR 2025 · similarity 0.89)
- Direct Consistency Optimization for Robust Customization of Text-to-Image Diffusion models (NeurIPS 2024 · similarity 0.89)
- LoRACLR: Contrastive Adaptation for Customization of Diffusion Models (CVPR 2025 · similarity 0.89)
- Orthogonal Adaptation for Modular Customization of Diffusion Models (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.