Research map / Continual learning / Online and replay-based
ConceptGuard: Continual Personalized Text-to-Image Generation with Forgetting and Confusion Mitigation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ConceptGuard, a continual personalized text-to-image method on SDXL that combines shift embeddings, concept-binding prompts, memory preservation regularization and a priority queue to mitigate catastrophic forgetting and concept confusion.
Problem
Sequential customization of diffusion models with few images per concept causes catastrophic forgetting of earlier concepts and confusion/blending when composing multiple concepts.
Approach
Shift embeddings adapt previously learned text embeddings to model updates; concept-binding prompts (with SAM-based concept focus and chrono-concept composition) and a priority queue of concept importance/order drive replay; a memory preservation regularizer on LoRA weight changes slows drift.
Key results
- ConceptGuard reaches 81.3 single-concept and 69.8 multi-concept image alignment, versus 77.5 and 57.1 for Continual Diffusion.
- Forgetting scores drop to FI 0.9 and FT 0.9 (image/text forgetting) versus 1.7 and 1.7-range for Continual Diffusion; multi-concept FI 1.9 vs 4.1.
- Ablation: removing concept-binding prompts causes the largest drop (multi-concept IA 59.3 vs 69.8).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ConceptGuard instead of Continual Diffusion | SDXL continual customization · Continual customization, multi-concept | Image alignment: 57.1 -> 69.8 | Helped |
| add shift embeddings (ablation) | ConceptGuard · Continual customization, multi-concept | Image alignment: 63.4 -> 69.8 | Helped |
| add priority queue (ablation) | ConceptGuard · Continual customization, multi-concept | Image alignment: 67.1 -> 69.8 | Helped |
| add concept-binding prompts (ablation) | ConceptGuard · Continual customization, multi-concept | Image alignment: 59.3 -> 69.8 | Helped |
| add memory preservation regularization (ablation) | ConceptGuard · Continual customization | FI: 2.7 -> 1.9 | Helped |
Surprising findings
- Textual Inversion, which keeps the diffusion model frozen, shows zero forgetting but lower alignment scores and fails in multi-concept generation.
Limitations
- Evaluation uses a single benchmark of 18 concepts and CLIP-based metrics only.
- Replay requires generating images of previous concepts with the model itself, and SAM-based concept focus adds a preprocessing dependency.
Experimental setup
Datasets: custom concept images (18 concepts: persons, pets, cartoon characters)
Benchmarks: custom concept continual customization benchmark (18 concepts, 6 per run)
Metrics: text alignment (CLIP) image alignment (CLIP) forgetting text (FT) forgetting image (FI)
Baselines: Textual Inversion DreamBooth Custom Diffusion Continual Diffusion EWC LwF
Builds on: Textual Inversion DreamBooth Custom Diffusion Continual Diffusion LoRA
Closest papers by meaning
- How to Continually Adapt Text-to-Image Diffusion Models for Flexible Customization? (NeurIPS 2024 · similarity 0.92)
- Concept Weaver: Enabling Multi-Concept Fusion in Text-to-Image Models (CVPR 2024 · similarity 0.87)
- Concept Conductor: Orchestrating Multiple Personalized Concepts in Text-to-Image Synthesis (AAAI 2025 · similarity 0.87)
- MC^2: Multi-concept Guidance for Customized Multi-concept Generation (CVPR 2025 · similarity 0.87)
- ClassDiffusion: More Aligned Personalization Tuning with Explicit Class Guidance (ICLR 2025 · similarity 0.87)
- Non-confusing Generation of Customized Concepts in Diffusion Models (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.