Research map / Image generation / Text-to-image diffusion
CoRe: Context-Regularized Text Embedding Learning for Text-to-Image Personalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CoRe, which learns a new concept's text embedding for Stable Diffusion personalization by regularizing the output embeddings and attention maps of surrounding context tokens against a super-category reference prompt, plus a two-stage embedding-then-U-Net training and optional test-time optimization.
Problem
Text-to-image personalization struggles to balance identity preservation with text alignment, because overfitted new-concept text embeddings corrupt the CLIP text encoder's representations of surrounding context tokens.
Approach
Regularize the context tokens rather than the new concept: enforce similarity between output embeddings and cross-attention maps of context tokens in a prompt with the new concept versus the same prompt with a super-category token, applied over a broad regularization prompt set without generating images, plus embedding norm rescaling.
Key results
- On quantitative evaluation, CoRe achieves CLIP-T 0.2568, CLIP-I 0.7054 and DINO 0.5842, the best CLIP-T and DINO among the compared methods (AttnDreamBooth 0.2547 / 0.6918 / 0.5641).
- In a user study with 1200 responses from 60 participants, CoRe is preferred over Custom Diffusion (85.7%), NeTI (76.3%), OFT (71.6%) and AttnDreamBooth (64.7%).
- Ablation: removing context attention regularization raises CLIP-I and DINO (0.7123 / 0.6060) but lowers CLIP-T (0.2486), indicating overfitting to the concept.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CoRe instead of AttnDreamBooth | Text-to-image personalization · 24 concepts, 20 prompts | CLIP-T: 0.2547 -> 0.2568 | Helped |
| Use CoRe instead of AttnDreamBooth | Text-to-image personalization · 24 concepts, 20 prompts | DINO: 0.5641 -> 0.5842 | Helped |
| Use CoRe instead of NeTI | Text-to-image personalization · 24 concepts, 20 prompts | CLIP-I: 0.7104 -> 0.7054 | Hurt |
| Add Context Embedding Regularization (ablation) | CoRe · 24 concepts, 20 prompts | CLIP-T: 0.2504 -> 0.2568 | Helped |
| Add Context Embedding Regularization (ablation) | CoRe · 24 concepts, 20 prompts | CLIP-I: 0.6996 -> 0.7054 | Helped |
| Add Context Embedding Regularization (ablation) | CoRe · 24 concepts, 20 prompts | DINO: 0.5796 -> 0.5842 | Helped |
| Add Context Attention Regularization (ablation) | CoRe · 24 concepts, 20 prompts | CLIP-T: 0.2486 -> 0.2568 | Helped |
| Add Context Attention Regularization (ablation) | CoRe · 24 concepts, 20 prompts | CLIP-I: 0.7123 -> 0.7054 | Hurt |
| Add Context Attention Regularization (ablation) | CoRe · 24 concepts, 20 prompts | DINO: 0.6060 -> 0.5842 | Hurt |
| Add embedding rescaling (ablation) | CoRe · 24 concepts, 20 prompts | CLIP-T: 0.2561 -> 0.2568 | Helped |
| Add embedding rescaling (ablation) | CoRe · 24 concepts, 20 prompts | CLIP-I: 0.6968 -> 0.7054 | Helped |
| Add embedding rescaling (ablation) | CoRe · 24 concepts, 20 prompts | DINO: 0.5805 -> 0.5842 | Helped |
Surprising findings
- Removing context attention regularization increases identity metrics while lowering text alignment, showing a trade-off in the regularization terms.
- Constraining the new concept directly against its super-category degraded identity preservation, so the method avoids that constraint.
Limitations
- Still struggles with difficult compositions involving the learned concept and other objects, partly inherited from the pretrained model.
- Quantitative metrics are unreliable for high-visual-variability prompts, so those are excluded from quantitative evaluation.
Experimental setup
Datasets: 24 concepts from previous personalization studies regularization prompt set (200 prompts total, 100 per group)
Benchmarks: 24 concepts with 20 quantitative text prompts
Metrics: CLIP-T CLIP-I DINO user preference rate
Baselines: Custom Diffusion NeTI OFT AttnDreamBooth Cross Initialization PhotoMaker Face2Diffusion ClassDiffusion
Builds on: Textual Inversion DreamBooth Custom Diffusion ClassDiffusion
Closest papers by meaning
- AttnDreamBooth: Towards Text-Aligned Personalized Text-to-Image Generation (NeurIPS 2024 · similarity 0.90)
- InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning (CVPR 2024 · similarity 0.88)
- Salient Concept-Aware Generative Data Augmentation (NeurIPS 2025 · similarity 0.87)
- Tailored Visions: Enhancing Text-to-Image Generation with Personalized Prompt Rewriting (CVPR 2024 · similarity 0.87)
- Preserving Clusters in Prompt Learning for Unsupervised Domain Adaptation (CVPR 2025 · similarity 0.86)
- Token-to-Token Alignment of Text Embeddings for Semantic Blending (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.