Research map / Image generation / Text-to-image diffusion
Customization Assistant for Text-to-Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds CAFE, a tuning-free customization assistant that pairs a Llama-2-13B MLLM (outputting text and CLIP-embedding tokens) with a Stable Diffusion 2 generator conditioned on CLIP and DINOv2 features, trained on a dataset bootstrapped via self-improvement distillation.
Problem
Single-image customized text-to-image generation usually requires test-time fine-tuning or yields weak tuning-free results, and existing systems only accept directive or descriptive prompts rather than ambiguous or interrogative chat-style input.
Approach
An LLM (Llama-2-13B-chat, frozen, with trained projections) infers user intent from chat input and emits a CLIP global embedding that conditions a diffusion model along with DINOv2 patch features; a self-improvement via distillation (SID) loop regenerates training data with the trained model, mixing reference-image embeddings to cover rare styles, to scale data cheaply without human filtering.
Key results
- On DreamBench CAFE reaches DINO 0.715, CLIP-I 0.827, CLIP-T 0.294 as a tuning-free method, competitive with fine-tuned baselines and above most tuning-free ones except SuTI on DINO.
- On the human face domain CAFE achieves ID 0.464 and CLIP-T 0.297, beating ProFusion (0.432 / 0.293) and E4T (0.426 / 0.277).
- Self-distillation data (D3, 148K samples at 1,500 GPU hours) yields ID 0.465 versus 0.441 for human-filtered data D2 (93K samples).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CAFE instead of SuTI | Tuning-free customization on DreamBench · DreamBench | DINO: 0.741 -> 0.715 | Hurt |
| Use CAFE instead of Kosmos-G | Tuning-free customization on DreamBench · DreamBench | CLIP-I: 0.847 -> 0.827 | Hurt |
| Use CAFE instead of DreamBooth | Customization on DreamBench · DreamBench | CLIP-T: 0.305 -> 0.294 | Hurt |
| Use CAFE instead of ProFusion | Human face customization · Human face domain | ID: 0.432 -> 0.464 | Helped |
| Use CAFE instead of ProFusion | Human face customization · Human face domain | CLIP-T: 0.293 -> 0.297 | Helped |
| Use self-generated samples with automatic filtering in addition to the initial data (ablation) | CAFE training on human face data · Human face domain | CLIP-T: 0.288 -> 0.297 | Helped |
| Use self-generated samples with automatic filtering in addition to the initial data (ablation) | CAFE training on human face data · Human face domain | ID: 0.441 -> 0.464 | Mixed |
| Replace mean squared error with negative cosine similarity (ablation) | CAFE embedding loss · Human face domain | ID: 0.464 -> 0.456 | Hurt |
| Replace mean squared error with negative cosine similarity (ablation) | CAFE embedding loss · Human face domain | CLIP-T: 0.297 -> 0.295 | Hurt |
| Replace CLIP ViT-L/14@336px with ViT-B/32 (ablation) | CAFE image encoder · Human face domain | CLIP-T: 0.297 -> 0.252 | Hurt |
| Replace DINOv2-Giant with DINOv2-Base (ablation) | CAFE image encoder · Human face domain | ID: 0.464 -> 0.408 | Hurt |
| Add DINO embedding to CLIP-only generation (ablation) | CAFE generation conditioning · Human face domain | ID: 0.216 -> 0.464 | Helped |
Surprising findings
- Training on the self-distilled dataset D3 without human filtering outperforms the human-filtered dataset D2 in identity similarity.
- Using only DINO embeddings gives good ID but poor CLIP-T, while using only CLIP gives poor ID, supporting the semantic/detail split.
Limitations
- Relies on ProFusion-generated targets and filtering by CLIP/DINO similarity, which may limit quality.
- Human face and DreamBench results compare against baselines whose numbers are copied from prior papers under differing prompts and base models (SuTI uses Imagen).
Experimental setup
Datasets: FFHQ DreamBench Stable Diffusion-generated object reference images ProFusion-generated targets
Benchmarks: DreamBench human face domain (7 researcher images, 23 prompts)
Metrics: DINO CLIP-I CLIP-T ID similarity
Baselines: Textual Inversion DreamBooth CustomDiffusion BLIP-Diffusion ELITE Subject-Diffusion SuTI Kosmos-G E4T ProFusion Re-Imagen
Builds on: LLaVA Stable Diffusion ProFusion DreamBooth Textual Inversion
Closest papers by meaning
- Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluation (ICLR 2024 · similarity 0.88)
- Dynamic Prompt Optimizing for Text-to-Image Generation (CVPR 2024 · similarity 0.87)
- LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Prompts (ICLR 2024 · similarity 0.87)
- LLM4GEN: Leveraging Semantic Representation of LLMs for Text-to-Image Generation (AAAI 2025 · similarity 0.87)
- FreeCustom: Tuning-Free Customized Image Generation for Multi-Concept Composition (CVPR 2024 · similarity 0.87)
- How to Continually Adapt Text-to-Image Diffusion Models for Flexible Customization? (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.