Research map / Image generation / Text-to-image diffusion
Diffusion Self-Distillation for Zero-Shot Customized Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses FLUX.1-dev to generate its own grid-based identity-consistent image pairs, curates them with a VLM (Gemini-1.5), and fine-tunes FLUX into a text+image-to-image model for zero-shot customization.
Problem
Zero-shot identity-preserving customized image generation is hard because there is no large-scale paired supervision for training image+text-conditioned diffusion models.
Approach
Exploit a text-to-image model's in-context ability to generate consistent image grids, curate same-identity pairs automatically with a VLM, and fine-tune the same model with a two-frame "video" architecture that treats the input image as the first frame, so the model learns supervised image-conditioned generation.
Key results
- On DreamBench++ with GPT-4o judging, the method achieves the best overall CP·PF product (0.631 for CP and 0.789 PF overall in the table format), trailing only IP-Adapter+ on concept preservation and DreamBooth-LoRA on prompt following.
- Human user study gives the method the highest creativity score (4.453) and PF 3.328, versus IP-Adapter+ CP 3.733.
- Ablation shows fine-tuning FLUX on grids via LoRA raises consistent-grid rate but reduces output diversity, motivating VLM curation instead.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Diffusion Self-Distillation instead of IP-Adapter+ | DreamBench++ image generation · DreamBench++ | de-biased concept preservation × prompt following: 0.187 vs 0.597 | Helped |
| Use Diffusion Self-Distillation instead of DreamBooth LoRA | DreamBench++ image generation · DreamBench++ | de-biased concept preservation × prompt following: 0.588 vs 0.597 | Helped |
| Use Diffusion Self-Distillation instead of DreamBooth LoRA | DreamBench++ image generation · DreamBench++ | prompt following overall: 0.849 vs 0.726 | Hurt |
| Use Diffusion Self-Distillation instead of IP-Adapter+ | DreamBench++ image generation · DreamBench++ | concept preservation overall: 0.834 vs 0.631 | Hurt |
| Fine-tune the base FLUX model on more than 7000 consistent grids using LoRA (ablation) | FLUX teacher grid generation · consistent image-grid generation | output diversity | Hurt |
| Curate diverse, potentially noisy grids with VLMs instead of using a consistent-grid LoRA (ablation) | Diffusion Self-Distillation data generation · training image-pair dataset | output diversity | Helped |
| Use the parallel processing architecture instead of concatenating the source image to the noise image (ablation) | Diffusion Self-Distillation · identity-preserving image generation | identity preservation | Helped |
| Use the parallel processing architecture instead of a ControlNet-based design (ablation) | Diffusion Self-Distillation · identity-preserving image generation | identity preservation | Helped |
| Use the parallel processing architecture instead of an IP-Adapter-based design (ablation) | Diffusion Self-Distillation · identity-preserving image generation | identity preservation | Helped |
| Use ControlNet for edits where source and target images differ in camera pose (ablation) | ControlNet-based image-to-image generation · identity-preserving image generation with camera-pose changes | detail preservation | Hurt |
| Use an IP-Adapter-based design for transferring source-image details and styles (ablation) | IP-Adapter-based image-to-image generation · image-to-image generation | detail and style transfer | Hurt |
Surprising findings
- Fine-tuning the teacher on consistent grids improves consistency but hurts diversity, so the authors rely on VLM filtering of noisy grids instead.
- IP-Adapter+ has the highest concept preservation score largely due to a copy-paste effect, which disappears under the debiased GPT evaluation.
Limitations
- Focuses on identity-preserving edits of characters, objects, and relighting; other tasks left for future work.
- Metrics (GPT-4o judging) are noisy and biased toward copy-paste; CLIP/DINO metrics judged unreliable by authors.
Experimental setup
Datasets: LAION captions (prompt references) self-generated FLUX image grids DreamBench++
Benchmarks: DreamBench++
Metrics: GPT-4o concept preservation score GPT-4o prompt following score debiased concept preservation debiased prompt following user study scores
Baselines: Textual Inversion DreamBooth DreamBooth LoRA BLIP-Diffusion Emu2 IP-Adapter IP-Adapter+
Builds on: FLUX.1-dev LoRA ControlNet IP-Adapter DreamBench++
Closest papers by meaning
- Guided Score identity Distillation for Data-Free One-Step Text-to-Image Generation (ICLR 2025 · similarity 0.91)
- SwiftBrush: One-Step Text-to-Image Diffusion Model with Variational Score Distillation (CVPR 2024 · similarity 0.89)
- D-OPSD: On-Policy Self-Distillation for Continuously Tuning Step-Distilled Diffusion Models (NeurIPS 2026 · similarity 0.89)
- R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation (ICLR 2024 · similarity 0.89)
- CoDi: Conditional Diffusion Distillation for Higher-Fidelity and Faster Image Generation (CVPR 2024 · similarity 0.89)
- Random Conditioning for Diffusion Model Compression with Distillation (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.