Research map / Image generation / Text-to-image diffusion
Generating Images of Rare Concepts Using Pre-trained Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SeedSelect, which optimizes the initial noise seed of a frozen Stable Diffusion model using a few reference images (CLIP semantic loss plus VAE latent appearance loss) to generate rare concepts and hands, and uses the generated images for few-shot data augmentation.
Problem
Text-to-image diffusion models fail to faithfully generate concepts under-represented in web-scale training data (long-tail concepts) and structured concepts such as hands.
Approach
Treat generation as per-class test-time optimization of the initial noise seed z_T, with no model fine-tuning, so that the diffusion model maps the seed to images matching a few reference images in both CLIP semantic space and VAE latent appearance.
Key results
- On LAION-tail ImageNet classes, SeedSelect reaches 76.1% accuracy on few-shot (tail) classes versus 52.2% for SD and 61.4% for Finetuned SD, with total accuracy 91.3%.
- FID of 6.5 for SeedSelect matches vanilla SD (6.4), while Finetuned SD degrades to 10.2.
- In hand-palm human evaluation, SeedSelect is preferred for matching the prompt at 70.21% versus 16.22% for SD.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Optimize the initial noise seed using reference images | Stable Diffusion v2.1 · ImageNet1k few-shot classes | accuracy: 52.2 -> 76.1 | Helped |
| Optimize the initial noise seed using reference images | Stable Diffusion v2.1 · CUB few-shot classes | accuracy: 52.0 -> 80.9 | Helped |
| Optimize the initial noise seed using reference images | Stable Diffusion v2.1 · iNaturalist few-shot classes | accuracy: 26.9 -> 68.8 | Helped |
| Use seed optimization instead of fine-tuning the diffusion model | Stable Diffusion v2.1 · ImageNet test images | FID: 6.4 -> 6.5 | No clear effect |
| Add reference-image guidance and filtering | Stable Diffusion v2.1 · ImageNet test images | FID: 6.4 -> 6.9 | Hurt |
| Fine-tune the diffusion model on class reference images | Stable Diffusion v2.1 · ImageNet test images | FID: 6.4 -> 10.2 | Hurt |
| Add contrasting-class contrastive loss | SeedSelect few-shot augmentation · miniImageNet and CIFAR-FS 5-way classification | accuracy: 86.01±0.7 -> 87.02±0.8; 91.01±0.8 -> 92.08±0.7; 91.12±0.5 -> 93.43±0.4; 93.46±0.4 -> 94.87±0.4 | Helped |
| Use SeedSelect-generated images for CLIP classifier fine-tuning | CLIP-RN50 · iNaturalist few-shot recognition | top-1 accuracy: 71.8±0.9 -> 74.8±0.6 | Helped |
| Use SeedSelect rather than Stable Diffusion for hand image generation | Stable Diffusion · Hand generation | rater preference for matching prompt (%): 16.22±2.6 -> 70.21±2.6 | Helped |
| Use SeedSelect rather than Stable Diffusion for hand image generation | Stable Diffusion · Hand generation | rater preference for realistic images (%): 16.19±5.8 -> 62.46±6.5 | Helped |
Surprising findings
- About 25% of ImageNet classes are poorly generated by Stable Diffusion, even though they have thousands of training samples in LAION2B.
- Finetuning SD on reference images for rare classes reduces image quality (FID 10.2) and diversity relative to vanilla SD.
Limitations
- Struggles to imitate the style of reference images, e.g. sketches often generate natural images.
- Optimized seed is prompt-specific and does not generalize directly to other prompts.
- For extremely rare concepts with very few LAION2B examples, generated image quality is poor.
Experimental setup
Datasets: ImageNet CUB-200 iNaturalist LAION2B LAION400M miniImageNet CIFAR-FS
Benchmarks: ImageNet CUB iNaturalist miniImageNet CIFAR-FS
Metrics: per-class accuracy with pretrained classifier FID Precision Recall Fidelity Diversity NDB human preference (2AFC) few-shot classification accuracy
Baselines: Stable Diffusion v2.1 SD+RG+RF Finetuned SD Textual Inversion CooP Tip-Adapter CT & SD DiffAlign SVAE SEGA Label-Hallucination FeLMi
Builds on: Stable Diffusion Real Guidance Real Filtering Textual Inversion CLIP
Closest papers by meaning
- Rare-to-Frequent: Unlocking Compositional Generation Power of Diffusion Models on Rare Concepts with LLM Guidance (ICLR 2025 · similarity 0.89)
- DreamDistribution: Learning Prompt Distribution for Diverse In-distribution Generation (ICLR 2025 · similarity 0.88)
- Fine-Grained Erasure in Text-to-Image Diffusion-based Foundation Models (CVPR 2025 · similarity 0.88)
- LLM4GEN: Leveraging Semantic Representation of LLMs for Text-to-Image Generation (AAAI 2025 · similarity 0.87)
- Effective Data Augmentation With Diffusion Models (ICLR 2024 · similarity 0.87)
- The Hidden Language of Diffusion Models (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.