Research map / Image generation / Text-to-image diffusion
Harmonizing Visual and Textual Embeddings for Zero-Shot Text-to-Image Customization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes orthogonal visual embedding and self-attention swap to fix pose bias and identity loss in zero-shot T2I customization methods such as BLIP-Diffusion and ELITE, without extra training.
Problem
Zero-shot text-to-image customization methods that encode a subject image into a visual embedding suffer from pose bias (generated subject keeps the input pose) and identity loss, due to conflict between visual and textual embeddings.
Approach
Orthogonalize the visual embedding against the subspace spanned by the textual embeddings (excluding the subject pseudo-word and class name) to remove pose-related interference, and add a second denoising pass guided by visual-only embedding whose self-attention keys/values are swapped into the subject region to restore identity.
Key results
- On the DS set, applying the full method to ELITE raises CLIP-T from 0.285 to 0.300 while M-CLIP-I stays at 0.837 and M-DINO-I at 0.581.
- On the DS set, the full method on BLIP-Diffusion raises CLIP-T from 0.262 to 0.272 with M-DINO-I rising from 0.617 to 0.630.
- In an Amazon Mechanical Turk study, ours was preferred for text alignment over BLIP-Diffusion (62.0% vs 38.0%) and over ELITE (56.3% vs 43.7%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add orthogonal visual embedding (contextual embedding orchestration) (ablation) | BLIP-Diffusion · Deformable Subject Set | CLIP-T: 0.262 -> 0.276 | Helped |
| add orthogonal visual embedding (contextual embedding orchestration) (ablation) | BLIP-Diffusion · Deformable Subject Set | CLIP-I: 0.853 -> 0.819 | Hurt |
| add self-attention swap (ablation) | BLIP-Diffusion · Deformable Subject Set | DINO-I: 0.684 -> 0.693 | Helped |
| add orthogonal visual embedding + self-attention swap (full method) | BLIP-Diffusion · Deformable Subject Set | CLIP-T: 0.262 -> 0.272 | Helped |
| add orthogonal visual embedding + self-attention swap (full method) | ELITE · Deformable Subject Set | CLIP-T: 0.285 -> 0.300 | Helped |
| add orthogonal visual embedding + self-attention swap (full method) | ELITE · Deformable Subject Set | DINO-I: 0.486 -> 0.502 | Helped |
| add orthogonal visual embedding + self-attention swap (full method) | BLIP-Diffusion · DreamBooth Set | CLIP-T: 0.295 -> 0.296 | No clear effect |
| add orthogonal visual embedding + self-attention swap (full method) | BLIP-Diffusion · Deformable Subject Set | CLIP-I: 0.853 -> 0.819 | Hurt |
Surprising findings
- Orchestration alone improves text alignment but lowers identity metrics (CLIP-I, DINO-I) for BLIP-Diffusion on the DreamBooth set, so identity loss needs the self-attention swap.
- Orthogonalizing against the input image's caption embedding does not resolve the conflict; only orthogonalizing against the generation prompt's text embedding does.
Limitations
- When multiple prompts with different instructions are given, the visual embedding cannot be orthogonalized to each textual embedding, so some instructions are omitted.
- Gains in identity metrics are mixed; for BLIP-Diffusion on the DS set the full method's CLIP-I and DINO-I are lower than the baseline.
Experimental setup
Datasets: DreamBooth dataset CustomConcept/CustomDiffusion subjects
Benchmarks: Deformable Subject Set (DS set) DreamBooth dataset
Metrics: CLIP-T CLIP-I DINO-I M-CLIP-I M-DINO-I LPIPS user preference
Baselines: BLIP-Diffusion ELITE
Builds on: BLIP-Diffusion ELITE Latent diffusion models Prompt-to-prompt cross-attention control
Closest papers by meaning
- MS-Diffusion: Multi-subject Zero-shot Image Personalization with Layout Guidance (ICLR 2025 · similarity 0.88)
- Decoupled Textual Embeddings for Customized Image Generation (AAAI 2024 · similarity 0.88)
- AttnDreamBooth: Towards Text-Aligned Personalized Text-to-Image Generation (NeurIPS 2024 · similarity 0.87)
- CoRe: Context-Regularized Text Embedding Learning for Text-to-Image Personalization (AAAI 2025 · similarity 0.86)
- Large-Scale Text-to-Image Model with Inpainting is a Zero-Shot Subject-Driven Image Generator (CVPR 2025 · similarity 0.86)
- CustomContrast: A Multilevel Contrastive Perspective for Subject-Driven Text-to-Image Customization (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.