Research map / Image generation / Editing and style transfer
Editable Noise Map Inversion: Encoding Target-image into Noise For High-Fidelity Image Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ENM Inversion, which refines DDIM noise maps at each inversion step to minimize the gap between source-prompt and target-prompt denoised noise, improving editability and preservation for diffusion-based image and video editing.
Problem
Inverting real images into noise maps for text-guided diffusion editing tends to reconstruct the source well but limits editability toward the target prompt, so edits adhering to the target text are weak.
Approach
Instead of optimizing noise maps only for source reconstruction, iteratively refine each inversion-step latent to minimize both a reconstruction loss and an editing-alignment loss that is the difference between source-conditioned and target-conditioned denoised noise maps, so the target is imprinted onto the noise.
Key results
- On PIE-Bench with P2P, ENM Inversion reaches structure distance 10.13e-3 and PSNR 28.19, versus 11.65 and 27.22 for the best prior baseline PNPInv, with edited CLIP similarity 22.12 close to the top of 22.44 for DDIM.
- With PnP editing, ENM Inversion improves PSNR from 22.46 (PNPInv) to 25.32 and structure distance from 24.29 to 18.44.
- On video editing with Video-P2P, ENM Inversion achieves CLIP score 26.57 versus 26.14 for NTI+Video-P2P, with LPIPS 98.24 versus 104.64.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ENM Inversion instead of PNPInv with P2P | P2P image editing · PIE-Bench | PSNR: 27.22 -> 28.19 | Helped |
| Use ENM Inversion instead of PNPInv with P2P | P2P image editing · PIE-Bench | CLIP similarity, edited: 22.10 -> 22.12; DDIM 22.44 -> 22.12 | Mixed |
| Use ENM Inversion instead of PNPInv with MasaCtrl | MasaCtrl image editing · PIE-Bench | PSNR: 22.64 -> 23.01 | Helped |
| Use ENM Inversion instead of PNPInv with PnP | PnP image editing · PIE-Bench | Structure distance (×10³): 24.29 -> 18.44 | Helped |
| Use ENM Inversion instead of NTI with Video-P2P | Video-P2P video editing · Video editing | LPIPS: 104.64 -> 98.24 | Helped |
| Use ENM Inversion instead of DDIM with TAV | Video editing · Video editing | TEMP: 0.9464 -> 0.9454 | Hurt |
| Increase edit-alignment weight λ (ablation) | ENM Inversion + P2P · PIE-Bench | CLIP similarity, whole: 25.17 -> 25.34 | Helped |
| Increase edit-alignment weight λ (ablation) | ENM Inversion + P2P · PIE-Bench | Structure distance (×10³): 10.10 -> 12.41 | Hurt |
| Increase DDIM sampling steps T (ablation) | ENM Inversion + P2P · PIE-Bench | CLIP similarity, whole: 24.95 -> 25.44 | Helped |
| Increase DDIM sampling steps T (ablation) | ENM Inversion + P2P · PIE-Bench | Structure distance (×10³): 8.74 -> 10.89 | Hurt |
| Use null text instead of the source prompt during inversion (ablation) | ENM Inversion · PIE-Bench reconstruction | PSNR: 28.19 -> 28.18 | No clear effect |
| Integrate ENM refinement into RFInv | RFInv · PIE-Bench | Structure distance (×10³): 60.08 -> 46.38 | Helped |
Surprising findings
- Fixed-point iteration methods (AIDI, FPI) improve structure and background preservation over DDIM but lower CLIP similarity, indicating a preservation-editability trade-off.
- Increasing the editing weight lambda or the number of sampling steps T improves CLIP similarity but degrades structure and background preservation.
Limitations
- Depends on the generative capacity of Stable Diffusion; target images outside its domain may not edit well.
- Requires a separate inversion per source image and target prompt, adding computational cost and inference time (about 38.87 s vs 18.22 s for DDIM).
Experimental setup
Datasets: PIE-Bench AFHQ DAVIS
Benchmarks: PIE-Bench DAVIS
Metrics: DINO structure distance PSNR LPIPS MSE SSIM CLIP similarity TEMP cross-attention alignment score
Baselines: DDIM Inversion Null-Text Inversion StyleDiffusion Noise Map Guidance Edit-Friendly DDPM Inversion PnP Inversion AIDI FPI ReNoise DDIM+TAV NTI+Video-P2P SDEdit-Flux
Builds on: DDIM inversion Null-Text Inversion PnP Inversion Prompt-to-Prompt Pix2pix-Zero
Closest papers by meaning
- Noise Map Guidance: Inversion with Spatial Context for Real Image Editing (ICLR 2024 · similarity 0.93)
- An Edit Friendly DDPM Noise Space: Inversion and Manipulations (CVPR 2024 · similarity 0.91)
- Inversion-Free Image Editing with Language-Guided Diffusion Models (CVPR 2024 · similarity 0.90)
- Schedule Your Edit: A Simple yet Effective Diffusion Noise Schedule for Image Editing (NeurIPS 2024 · similarity 0.90)
- Object-Aware Inversion and Reassembly for Image Editing (ICLR 2024 · similarity 0.89)
- PnP Inversion: Boosting Diffusion-based Editing with 3 Lines of Code (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.