Research map / Image generation / Text-to-image diffusion
DreamSteerer: Enhancing Source Image Conditioned Editability using Personalized Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DreamSteerer, a plug-in that improves source-image-conditioned editability of existing T2I personalization models (Textual Inversion, DreamBooth, Custom Diffusion) via Editability Driven Score Distillation with mode-shifting regularization built from spatial-feature-guided samples.
Problem
Personalized T2I models lose editability when used to edit real source images: they collapse to reference patterns, lose prior knowledge of the source class, and fail to preserve source structure under significant structural changes.
Approach
Fine-tune the personalized model parameters with an editability-driven score distillation loss that distills the source model's score on the source image into the personalized model via a single-step perturbed source latent, and avoid mode trapping by regularizing with DPM loss on samples guided by self-attention features of the source image (PatchNCE classifier guidance).
Key results
- On DreamBooth, DreamSteerer lowers LPIPS (Alex) from 0.204 to 0.183 and raises CLIP-I L/14 from 0.751 to 0.761 over the baseline.
- In user preference (1-5), DreamSteerer scores 3.78 vs 2.48 for Textual Inversion, 3.76 vs 2.90 for DreamBooth, and 3.78 vs 2.62 for Custom Diffusion.
- Under one-shot personalization, DreamBooth LPIPS improves from 0.222 to 0.160 with DreamSteerer, with CLIP-I L/14 rising from 0.717 to 0.748.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add DreamSteerer to Textual Inversion | Textual Inversion · Personalized editing | LPIPS (Alex): 0.133 -> 0.115 | Helped |
| Add DreamSteerer to DreamBooth | DreamBooth · Personalized editing | SSIM: 0.755 -> 0.768 | Helped |
| Add DreamSteerer to Custom Diffusion | Custom Diffusion · Personalized editing | LPIPS (Alex): 0.199 -> 0.182 | Helped |
| Add DreamSteerer to DreamBooth | DreamBooth · One-shot personalized editing | LPIPS (Alex): 0.222 -> 0.160 | Helped |
| Add DreamSteerer to Textual Inversion | Textual Inversion · Personalized editing | User preference: 2.48 -> 3.78 | Helped |
| Add EDSD to the baseline while retaining Mode Shifting (ablation) | Textual Inversion · Personalized editing | SSIM: 0.821 -> 0.833 | Hurt |
| Add EDSD to the baseline while retaining Mode Shifting (ablation) | DreamBooth · Personalized editing | SSIM: 0.753 -> 0.768 | Helped |
| Add EDSD to the baseline while retaining Mode Shifting (ablation) | Custom Diffusion · Personalized editing | SSIM: 0.748 -> 0.779 | Helped |
| Add Mode Shifting regularization while retaining EDSD (ablation) | Textual Inversion · Personalized editing | CLIP-I (CLIP B/32): 0.779 -> 0.788 | Helped |
| Add Mode Shifting regularization while retaining EDSD (ablation) | DreamBooth · Personalized editing | CLIP-I (CLIP B/32): 0.782 -> 0.796 | Helped |
| Add Mode Shifting regularization while retaining EDSD (ablation) | Custom Diffusion · Personalized editing | CLIP-I (CLIP B/32): 0.760 -> 0.784 | Helped |
| Use a relatively small CFG of 3.5 rather than large CFG values such as 7 or 15 (ablation) | Spatial feature guided sampling · Generated sample naturalness | Naturalness | Helped |
Surprising findings
- Setting the Jacobian of the personalized denoiser to identity (as in prior work) destroyed source layout and oversaturated images; using -I worked better.
- Prompt-to-Prompt with Null-Text Inversion produced significant artifacts when combined with DreamBooth, because personalization shifts the source-class distribution and compromises inversion.
Limitations
- Performance is bounded by the baseline personalization model; e.g., a DreamBooth duck-toy model loses subject appearance after editing.
- Automatic metrics do not fully reflect perceived edit quality, so user studies were needed.
- Jacobian-term handling (-I) is an empirical choice without theoretical justification.
Experimental setup
Datasets: ViCo personalization checkpoints (16 concepts) 70 random real-world source images per baseline PIE-Bench
Benchmarks: ViCo-based editing set (16 concepts, 70 source images per baseline)
Metrics: CLIP-I (ViT-B/32, ViT-L/14) LPIPS (Alex, VGG) SSIM MS-SSIM Topiq Musiq LIQE user preference (1-5)
Baselines: Textual Inversion DreamBooth Custom Diffusion Custom-Edit DreamEdit Photoswap
Builds on: Score Distillation Sampling Delta Denoising Score DreamBooth Textual Inversion Custom Diffusion
Closest papers by meaning
- PaRa: Personalizing Text-to-Image Diffusion via Parameter Rank Reduction (ICLR 2025 · similarity 0.90)
- Personalized Image Editing in Text-to-Image Diffusion Models via Collaborative Direct Preference Optimization (NeurIPS 2025 · similarity 0.89)
- Energy-Guided Optimization for Personalized Image Editing with Pretrained Text-to-Image Diffusion Models (AAAI 2025 · similarity 0.89)
- Exploring Low-Dimensional Subspace in Diffusion Models for Controllable Image Editing (NeurIPS 2024 · similarity 0.87)
- DreamCatalyst: Fast and High-Quality 3D Editing via Controlling Editability and Identity Preservation (ICLR 2025 · similarity 0.87)
- DreamDistribution: Learning Prompt Distribution for Diverse In-distribution Generation (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.