Research map / Image generation / Text-to-image diffusion
DreamDistribution: Learning Prompt Distribution for Diverse In-distribution Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Learns a Gaussian distribution over soft prompt embeddings of Stable Diffusion 2.1 from a set of reference images, so sampled prompts yield diverse in-distribution image generations.
Problem
Personalizing text-to-image diffusion models at an abstract concept or category level from a set of reference images sharing commonalities, where existing instance-level personalization methods yield limited diversity or overfit.
Approach
Model the reference image set with a distribution over learnable soft prompt embeddings (mean and variance estimated from K prompts, sampled via reparameterization), trained with an orthogonality loss, keeping the diffusion model frozen so the learned prompts stay in the text embedding space.
Key results
- On the custom evaluation set, DreamDistribution reports CLIP-I 0.84, DINO 0.50, Density 1.59, Coverage 0.93 and FID 215.15, beating DreamBooth, Textual Inversion and Custom Diffusion on all metrics in the naive-adaptation comparison.
- On the French Bulldog granularity set, the method reaches DINO 0.61, Density 0.23, Coverage 0.34 and FID 88.81 versus the best baseline Textual Inversion FID 98.76.
- In the 3D diversity test, pairwise similarity drops from 0.63 (Textual Inversion) to 0.53 CLIP-I and from 0.31 to 0.14 DINO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DreamDistribution instead of Textual Inversion for 3D generation | MVDream · 3D generation | CLIP-I pairwise similarity: 0.63 -> 0.53 | Helped |
| Use DreamDistribution instead of Textual Inversion for 3D generation | MVDream · 3D generation | DINO pairwise similarity: 0.31 -> 0.14 | Helped |
| Increase the number of learned prompts (ablation) | DreamDistribution · Personalized generation | generation quality and diversity | Helped |
| Vary orthogonal-loss weight λ (ablation) | DreamDistribution · Personalized generation | quantitative metrics | Mixed |
| Increase the number of token vectors per prompt (ablation) | DreamDistribution · Personalized generation | generation quality and diversity | Helped |
| Increase the number of token vectors per prompt (ablation) | DreamDistribution · Personalized generation | training time | Hurt |
| Train the prompt distribution using 100–500 images per class rather than all available training images (ablation) | DreamDistribution synthetic ImageNet data · ImageNet validation | classification accuracy | Helped |
| Add more synthetic images to the real training set (ablation) | ImageNet classifier trained on DreamDistribution data · ImageNet validation | Top-1 and Top-5 accuracy | Helped |
| Add more synthetic images to the real training set (ablation) | ImageNet classifier trained on DreamDistribution data · ImageNet variant validation sets | accuracy | No clear effect |
| Use DreamDistribution rather than personalization baselines | Personalized generation · French Bulldog | FID: 100.24 -> 88.81 | Helped |
| Use DreamDistribution rather than personalization baselines | Personalized generation · Dogs and Quadrupeds | feature-based metrics | Mixed |
Surprising findings
- On the coarser-grained Dogs and Quadrupeds sets the advantage over personalization baselines largely disappears, and on the DreamBooth same-instance task the method is only on par (CLIP-I 0.80, DINO 0.59 vs DreamBooth 0.67).
- On ImageNet variants (IN-V2, IN-S, IN-R, IN-A), using fewer training images per class often gives higher accuracy, attributed to domain gap rather than synthetic data quality.
Limitations
- Struggles to capture visual features when the number of training images is limited and very diverse.
- The Gaussian distribution assumption over prompt embeddings may be overly restrictive.
- On coarser-grained reference sets (Dogs, Quadrupeds), advantage diminishes and outliers can be sampled.
Experimental setup
Datasets: DreamBooth dataset ImageNet ImageNet-V2 ImageNet-Sketch ImageNet-R ImageNet-A
Benchmarks: DreamBooth dataset ImageNet ImageNet-V2 ImageNet-Sketch ImageNet-R ImageNet-A
Metrics: CLIP-I CLIP-T DINO Density Coverage FID top-1 accuracy top-5 accuracy pairwise cosine similarity
Baselines: Textual Inversion DreamBooth Custom Diffusion
Builds on: prompt tuning Textual Inversion DreamBooth Custom Diffusion MVDream
Closest papers by meaning
- Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.93)
- Minority-Focused Text-to-Image Generation via Prompt Optimization (CVPR 2025 · similarity 0.92)
- Prompting Hard or Hardly Prompting: Prompt Inversion for Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.90)
- T2I-Adapter: Learning Adapters to Dig Out More Controllable Ability for Text-to-Image Diffusion Models (AAAI 2024 · similarity 0.90)
- Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs (ICML 2024 · similarity 0.90)
- Textualize Visual Prompt for Image Editing via Diffusion Bridge (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.