Research map / Image generation / Editing and style transfer
Energy-Guided Optimization for Personalized Image Editing with Pretrained Text-to-Image Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EGO-Edit, a training-free, inversion-free method that optimizes the latent image under text and feature-level image energy guidance from SD 1.5, with coarse-to-fine timestep scheduling and latent content composition for personalized object replacement.
Problem
Personalized image editing that replaces objects with reference-image objects, especially cross-class replacements with large domain gaps, where mask-as-shape priors and inversion-based noise initialization yield seamless-integration and identity-consistency failures.
Approach
Formulates personalized editing as energy-guided optimization of the edited latent, using the pretrained diffusion model itself as the energy function (text energy early, point-to-point feature image energy later), plus gradient scaling and latent-space copy-paste content composition, with no inversion or training.
Key results
- On customized object swapping, EGO-Edit reaches DINO 62.749, CLIP-T 28.764, CLIP-I 76.624, beating Anydoor (DINO 57.093) and Dragon (DINO 48.176).
- Ablation: removing image-energy guidance drops DINO from 62.749 to 56.510; removing content composition drops it to 57.531.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EGO-Edit instead of the strongest reported baseline, Anydoor | Customized image swapping · Customized image swapping | DINO: 57.093 -> 62.749 | Helped |
| Use EGO-Edit instead of the strongest reported baseline, PbE | Customized image swapping · Customized image swapping | CLIP-T: 27.149 -> 28.764 | Helped |
| Use EGO-Edit instead of the strongest reported baseline, Anydoor | Customized image swapping · Customized image swapping | CLIP-I: 72.133 -> 76.624 | Helped |
| Add image-energy guidance (IEG) (ablation) | EGO-Edit · Customized image swapping | DINO: 56.510 -> 62.749 | Helped |
| Add image-energy guidance (IEG) (ablation) | EGO-Edit · Customized image swapping | CLIP-I: 75.664 -> 76.624 | Helped |
| Add content composition (CC) (ablation) | EGO-Edit · Customized image swapping | DINO: 57.531 -> 62.749 | Helped |
| Add content composition (CC) (ablation) | EGO-Edit · Customized image swapping | CLIP-I: 75.457 -> 76.624 | Helped |
| Add gradient scaling (GS) (ablation) | EGO-Edit · Customized image swapping | editing quality | Helped |
| Apply content composition at a medium optimization stage rather than early or late (ablation) | EGO-Edit · Customized image swapping | editing quality | Mixed |
| Use a descending rather than random or ascending timestep schedule (ablation) | EGO-Edit · Customized image swapping | editing quality | Helped |
Limitations
- Negative prompts can trade quality for fidelity and introduce artifacts, so a smaller weight is used.
- Text prompts are ambiguous, so text energy alone cannot give precise guidance for personalized details.
- Content composition timing is sensitive: too early over-adheres to the reference, too late leaves blur.
Experimental setup
Datasets: DreamEditBench PIE-Bench
Benchmarks: DreamEditBench PIE-Bench
Metrics: DINO score CLIP-T CLIP-I user study
Baselines: Paint-by-Example Anydoor PAIR Dragon Graphit
Builds on: Score Distillation Sampling Delta Denoising Score DIFT IP-Adapter Classifier-free guidance
Closest papers by meaning
- Personalized Image Editing in Text-to-Image Diffusion Models via Collaborative Direct Preference Optimization (NeurIPS 2025 · similarity 0.90)
- PaRa: Personalizing Text-to-Image Diffusion via Parameter Rank Reduction (ICLR 2025 · similarity 0.90)
- DreamSteerer: Enhancing Source Image Conditioned Editability using Personalized Diffusion Models (NeurIPS 2024 · similarity 0.89)
- Direct Consistency Optimization for Robust Customization of Text-to-Image Diffusion models (NeurIPS 2024 · similarity 0.89)
- Inversion-Free Image Editing with Language-Guided Diffusion Models (CVPR 2024 · similarity 0.88)
- Invertible Consistency Distillation for Text-Guided Image Editing in Around 7 Steps (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.