Research map / Image generation / Text-to-image diffusion
HyperDreamBooth: HyperNetworks for Fast Personalization of Text-to-Image Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HyperDreamBooth, a hypernetwork (ViT-H encoder + transformer decoder) that predicts compact LoRA-style personalization weights (LiDB) from one face image, followed by ~40 steps of rank-relaxed fast finetuning, for fast face personalization of Stable Diffusion.
Problem
DreamBooth-style text-to-image personalization is slow (minutes per subject) and produces large per-subject models (over 1GB), limiting practical use.
Approach
A hypernetwork with a ViT image encoder and an iterative transformer decoder predicts Lightweight DreamBooth weights (a random orthogonal incomplete basis inside rank-1 LoRA space, ~30K variables, ~120KB) from a single face image; these are then refined with rank-relaxed LoRA finetuning for about 40 iterations.
Key results
- Personalization in roughly 20 seconds, 25x faster than DreamBooth and 125x faster than Textual Inversion, with the customized model about 10,000x smaller than a DreamBooth model.
- Table 1: face recognition 0.655 vs 0.618 (DreamBooth) and 0.623 (Textual Inversion); DINO 0.473 vs 0.441 and 0.289.
- User study: 64.8% preferred the method over DreamBooth for identity and 70.6% over Textual Inversion; 60.0% preferred it over E4T for identity and style.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HyperDreamBooth instead of DreamBooth | Stable Diffusion 1.5 face personalization · 100 CelebAHQ identities, 30 prompts | DINO: 0.441 -> 0.473 | Helped |
| Use HyperDreamBooth instead of DreamBooth | Stable Diffusion 1.5 face personalization · 100 CelebAHQ identities, 30 prompts | CLIP-T: 0.282 -> 0.286 | Helped |
| Remove the HyperNetwork at test time (ablation) | HyperDreamBooth · HyperDreamBooth ablation | DINO: 0.473 -> 0.392 | Hurt |
| Remove the HyperNetwork at test time (ablation) | HyperDreamBooth · HyperDreamBooth ablation | CLIP-T: 0.286 -> 0.299 | Helped |
| Use HyperNetwork prediction without fast finetuning (ablation) | HyperDreamBooth · HyperDreamBooth ablation | DINO: 0.473 -> 0.414 | Hurt |
| Disable iterative prediction (k=1) (ablation) | HyperDreamBooth · HyperDreamBooth ablation | DINO: 0.473 -> 0.464 | Hurt |
| Reduce DreamBooth training to 400 iterations | DreamBooth · DreamBooth comparison | DINO: 0.441 -> 0.323 | Hurt |
| Reduce DreamBooth training to 40 iterations | DreamBooth · DreamBooth comparison | DINO: 0.441 -> 0.360 | Hurt |
Surprising findings
- Aggressive DreamBooth training with fewer iterations or higher learning rates degraded results, showing the hypernetwork initialization matters.
- Removing the hypernetwork (No Hyper) gave slightly better CLIP-T prompt following than the full method.
Limitations
- Face recognition metrics are weak due to network realism bias, so the authors relied on user studies.
- Evaluated mainly on faces; generality to other subjects is not shown in the text.
Experimental setup
Datasets: CelebA-HQ SFHQ
Benchmarks: CelebA-HQ identities with 30 prompts (30,000 samples)
Metrics: Face Recognition similarity DINO CLIP-I CLIP-T user preference
Baselines: DreamBooth Textual Inversion E4T LoRA DreamBooth InstantBooth
Builds on: DreamBooth LoRA Stable Diffusion Textual Inversion E4T
Closest papers by meaning
- InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning (CVPR 2024 · similarity 0.91)
- PortraitBooth: A Versatile Portrait Model for Fast Identity-preserved Personalization (CVPR 2024 · similarity 0.87)
- JeDi: Joint-Image Diffusion Models for Finetuning-Free Personalized Text-to-Image Generation (CVPR 2024 · similarity 0.87)
- When StyleGAN Meets Stable Diffusion: a W+ Adapter for Personalized Image Generation (CVPR 2024 · similarity 0.86)
- AttnDreamBooth: Towards Text-Aligned Personalized Text-to-Image Generation (NeurIPS 2024 · similarity 0.86)
- Dis²Booth: Learning Image Distribution with Disentangled Features for Text-to-Image Diffusion Models (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.