Research map / Image generation / Text-to-image diffusion
InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes InstantBooth, a personalized text-to-image method that needs no test-time finetuning, using a CLIP-based concept encoder, patch encoder with gated adapter layers on Stable Diffusion, trained only on text-image pairs.
Problem
Personalized text-to-image generation usually requires costly per-concept test-time finetuning, which is slow, storage-heavy, and hard to scale; encoder-based alternatives often lose identity detail or language alignment.
Approach
Map input images to a global concept token in the text embedding space plus rich CLIP patch tokens fed through gated self-attention adapter layers in the U-Net, trained on single images with masked-background condition images (no concept-specific paired data), and at inference rebalance adapter strength (balanced sampling) and rescale the concept token (concept renormalization) to preserve languag
Key results
- On person category, InstantBooth achieves alignment 0.3140 and face similarity 0.2456, versus DreamBooth 0.3088 and 0.2408, with reconstruction lower (0.7329 vs 0.8335).
- Inference takes about 6 seconds per concept versus about 600 s for DreamBooth and about 1500 s for Textual Inversion, a roughly 100x speedup.
- User study (1094 valid samples) rates InstantBooth highest on quality (3.53), alignment (3.58) and identity (3.55) versus DreamBooth (3.50, 3.50, 3.55).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use InstantBooth instead of DreamBooth | Person personalization · Person test set | Align: 0.3088 -> 0.3140 | Helped |
| Use InstantBooth instead of DreamBooth | Person personalization · Person test set | Recon: 0.8335 -> 0.7329 | Hurt |
| Use InstantBooth instead of DreamBooth | Person personalization · Person test set | Time (s): ∼600 -> 6 | Helped |
| Remove object masking during training (ablation) | InstantBooth · Person test set | Align: 0.3140 -> 0.3127 | Hurt |
| Remove object masking during training (ablation) | InstantBooth · Person test set | Reconstruct: 0.7329 -> 0.7485 | Helped |
| Remove patch features, retaining only the global [CLS] token (ablation) | InstantBooth · Person test set | Align: 0.3140 -> 0.3269 | Helped |
| Remove patch features, retaining only the global [CLS] token (ablation) | InstantBooth · Person test set | Reconstruct: 0.7329 -> 0.6494 | Hurt |
| Remove the adapter branch (ablation) | InstantBooth · Person test set | Reconstruct: 0.7329 -> 0.5468 | Hurt |
| Insert the placeholder token before the CLIP text encoder (ablation) | InstantBooth · Person test set | Align: 0.3140 -> 0.3127 | Hurt |
| Tune the CLIP visual encoder (ablation) | InstantBooth · Person test set | Reconstruct: 0.7329 -> 0.6425 | Hurt |
| Tune the U-Net (ablation) | InstantBooth · Person test set | Align: 0.3140 -> 0.3142 | No clear effect |
| Use one input image instead of multiple images (ablation) | InstantBooth · Person test set | Reconstruct: 0.7329 -> 0.7261 | Hurt |
Surprising findings
- Higher reconstruction score from the pose-copying baselines does not translate to better identity preservation; the authors argue face similarity is the more robust identity metric.
- Removing the patch features or the adapter improves text alignment slightly but sharply degrades reconstruction, showing a trade-off between alignment and identity.
Limitations
- The adapter structure can only take a single concept as input at a time.
- Reconstruction score is lower than finetuning baselines because the model changes pose and background instead of copying them.
- Evaluated only on person and cat categories.
Experimental setup
Datasets: collected text-image pairs with entity segmentation masks (person and cat categories) PPR10K person dataset (test split, 50 identities)
Benchmarks: PPR10K person test subset
Metrics: CLIP alignment face cosine similarity CLIP reconstruction similarity inference time user study ratings (quality, alignment, identity)
Baselines: Textual Inversion DreamBooth ELITE
Builds on: Stable Diffusion GLIGEN DreamBooth Textual Inversion CLIP
Closest papers by meaning
- HyperDreamBooth: HyperNetworks for Fast Personalization of Text-to-Image Models (CVPR 2024 · similarity 0.91)
- AttnDreamBooth: Towards Text-Aligned Personalized Text-to-Image Generation (NeurIPS 2024 · similarity 0.90)
- JeDi: Joint-Image Diffusion Models for Finetuning-Free Personalized Text-to-Image Generation (CVPR 2024 · similarity 0.89)
- CoRe: Context-Regularized Text Embedding Learning for Text-to-Image Personalization (AAAI 2025 · similarity 0.88)
- PortraitBooth: A Versatile Portrait Model for Fast Identity-preserved Personalization (CVPR 2024 · similarity 0.88)
- Yo'Chameleon: Personalized Vision and Language Generation (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.