Research map / Image generation / Editing and style transfer
DiffEditor: Boosting Accuracy and Flexibility on Diffusion-based Image Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiffEditor, a training-light diffusion image editing method that adds an image-prompt encoder (trained on LAION) plus regional SDE sampling, regional score-based gradient guidance, and time travel on top of DragonDiffusion for fine-grained editing.
Problem
Diffusion-based fine-grained image editing suffers from low editing accuracy, artifacts in complex scenes, and limited flexibility (e.g., imagining new content) due to deterministic ODE sampling, text-only prompts, and strong content-consistency constraints.
Approach
Introduce an image prompt encoder (CLIP + QFormer with 64 queries, trained with the diffusion loss) alongside text for editing; inject randomness via regional SDE only in edit areas; localize editing and consistency gradients with the edit mask; and use time travel (DDIM-inversion rollbacks) during guidance steps.
Key results
- On CelebA-HQ 68-point face dragging, DiffEditor reaches MSE 11.52 and FID 33.02/33.10 versus DragonDiff's 13.94 and 34.58/35.75, with inference 13.88s vs 15.00s.
- Object pasting CLIP score 0.274 vs DragonDiff 0.260 and Paint-by-example 0.265; moving 0.288 vs 0.282; replacing 0.281 vs 0.263 (16 samples per task).
- Ablations show image prompt, regional gradient guidance, and time travel each reduce distortions qualitatively, though no numeric ablation table is given.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the DiffEditor image prompt encoder and image prompt conditioning | DiffEditor · Object pasting | CLIP distance: 0.260 -> 0.274 | Helped |
| Add the DiffEditor image prompt encoder and image prompt conditioning | DiffEditor · Object moving | CLIP distance: 0.282 -> 0.288 | Helped |
| Add the DiffEditor image prompt encoder and image prompt conditioning | DiffEditor · Appearance replacing | CLIP distance: 0.263 -> 0.281 | Helped |
| Add DiffEditor’s editing pipeline | Stable Diffusion 1.5 · CelebA-HQ, 17-point face manipulation | MSE distance: 18.51 -> 17.05 | Helped |
| Add DiffEditor’s editing pipeline | Stable Diffusion 1.5 · CelebA-HQ, 68-point face manipulation | MSE distance: 13.94 -> 11.52 | Helped |
| Use the image prompt rather than text prompting for DDIM inversion and editing (ablation) | DiffEditor · Image reconstruction and fine-grained editing | reconstruction/editing quality | Helped |
| Replace the image prompt encoder with IP-Adapter-plus (ablation) | DiffEditor · Content dragging | texture-detail consistency | Hurt |
| Apply editing gradients only within the editing-region mask (ablation) | DiffEditor · Fine-grained image editing | content consistency outside the edited region | Helped |
| Remove the regional constraint on the editing gradient (ablation) | DiffEditor · Fine-grained image editing | content consistency outside the edited region | Hurt |
| Add accurate time travel using deterministic DDIM inversion (ablation) | DiffEditor · Complex image-editing scenarios | editing quality | Helped |
| Remove time travel (ablation) | DiffEditor · Complex image-editing scenarios | editing quality | Hurt |
| Replace accurate time travel with random-noise time travel (ablation) | DiffEditor · Complex image-editing scenarios | content consistency | Hurt |
Surprising findings
- Reducing content-consistency strength (random initial noise or removing visual cross-attention) in DragonDiff improves flexibility but severely compromises consistency, motivating localized randomness.
Limitations
- Struggles with edits requiring large content imagination or 3D perception, e.g., rotating a car by dragging its front, attributed to Stable Diffusion's lack of 3D awareness.
- Evaluation on object moving/pasting/replacing uses only 16 samples per task with CLIP score.
Experimental setup
Datasets: LAION CelebA-HQ
Benchmarks: CelebA-HQ face manipulation (17/68 points)
Metrics: MSE landmark distance FID CLIP score runtime
Baselines: UserControllableLT DragGAN DragDiff DragonDiff Paint-by-example Self-Guidance IP-Adapter-plus
Builds on: DragonDiffusion IP-Adapter DDIM DragGAN Stable Diffusion
Closest papers by meaning
- Towards Efficient Diffusion-Based Image Editing with Instant Attention Masks (AAAI 2024 · similarity 0.87)
- FunEditor: Achieving Complex Image Edits via Function Aggregation with Diffusion Models (AAAI 2025 · similarity 0.86)
- High-Fidelity Diffusion-Based Image Editing (AAAI 2024 · similarity 0.85)
- Diff-Plugin: Revitalizing Details for Diffusion-based Low-level Tasks (CVPR 2024 · similarity 0.85)
- The Blessing of Randomness: SDE Beats ODE in General Diffusion-based Image Editing (ICLR 2024 · similarity 0.85)
- Stable Flow: Vital Layers for Training-Free Image Editing (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.