Research map / Image generation / Editing and style transfer
Inversion-Free Image Editing with Language-Guided Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes InfEdit, an inversion-free text-guided image editing method built on a Denoising Diffusion Consistent Model (virtual inversion) plus a tuning-free Unified Attention Control, evaluated on PIE-Bench with Stable Diffusion 1.4 and LCM backbones.
Problem
Text-guided image editing with diffusion models relies on time-consuming DDIM-style inversion, struggles to balance consistency and editing accuracy, and is incompatible with efficient consistency-model sampling.
Approach
Shows that with a known source latent, a special variance schedule makes the denoising step match multi-step consistency sampling (DDCM), yielding a virtual inversion with no explicit inversion; it calibrates the target initial prediction via a dual-branch noise difference and unifies cross-attention and mutual self-attention control through an extra layout branch.
Key results
- On PIE-Bench, VI+UAC with LCM reaches 2.22s forward time over 12 steps, versus about 10.9s for DDIM+P2P with 50 steps on one A40.
- VI with P2P achieves a higher PSNR (22.03 whole-image background metric-type comparison) than DDIM+P2P (21.69 vs 22.44 pattern) and comparable to inversion baselines, while being far faster.
- Table 2 shows LCM backbone gives higher CLIP scores than SD1.4 at fewer steps (25.76 at 2 steps vs 21.47 for SD1.4).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace P2P with UAC (ablation) | InfEdit with LCM on PIE-Bench · PIE-Bench | PSNR (10^3): 26.64 -> 28.51 | Helped |
| Replace P2P with UAC (ablation) | InfEdit with LCM on PIE-Bench · PIE-Bench | CLIP similarity, Whole (10^2): 24.57 -> 25.03 | Helped |
| Replace P2P with UAC (ablation) | InfEdit with LCM on PIE-Bench · PIE-Bench | Forward time (sec): 2.60 ± 0.00 -> 2.22 ± 0.02 | Helped |
| Use LCM instead of SD 1.4 (ablation) | InfEdit (VI+P2P), 2 sampling steps · PIE-Bench | CLIP Score: 21.47 -> 25.76 | Helped |
| Use LCM instead of SD 1.4 (ablation) | InfEdit (VI+P2P), 4 sampling steps · PIE-Bench | CLIP Score: 22.17 -> 25.33 | Helped |
| Use LCM instead of SD 1.4 (ablation) | InfEdit (VI+P2P), 8 sampling steps · PIE-Bench | CLIP Score: 23.16 -> 24.76 | Helped |
| Use LCM instead of SD 1.4 (ablation) | InfEdit (VI+P2P), 16 sampling steps · PIE-Bench | CLIP Score: 24.45 -> 25.08 | Helped |
| Use LCM instead of SD 1.4 (ablation) | InfEdit (VI+P2P), 32 sampling steps · PIE-Bench | CLIP Score: 25.18 -> 26.68 | Helped |
| Increase SD 1.4 sampling steps from 2 to 32 (ablation) | InfEdit (VI+P2P) · PIE-Bench | CLIP Score: 21.47 -> 25.18 | Helped |
| Increase LCM sampling steps from 2 to 32 (ablation) | InfEdit (VI+P2P) · PIE-Bench | CLIP Score: 25.76 -> 26.68 | Mixed |
| Use inversion-free VI instead of DDIM inversion | Image editing with P2P on PIE-Bench · PIE-Bench | Forward time (sec): 12.79 ± 0.01 -> 4.50 ± 0.01 | Helped |
| Use UAC rather than P2P for attention control | InfEdit on PIE-Bench editing categories · PIE-Bench, 9 editing tasks | Editing quality (CLIP Score) | Helped |
Surprising findings
- Inversion-free sampling with virtual inversion matches or surpasses several inversion-based baselines on PIE-Bench consistency metrics despite skipping inversion entirely.
- Some dual-branch methods (DI, CycleDiffusion) achieve lower structure distance than InfEdit but often fail to apply the requested edit.
Limitations
- Attention-control hyperparameters (cross-attention and self-attention switch steps, thresholds) must be set manually
- Reliance on pretrained SD1.4 or LCM backbones limits editing to what these models generate well
Experimental setup
Datasets: PIE-Bench Horse2Zebra Summer2Winter
Benchmarks: PIE-Bench Horse2Zebra Summer2Winter
Metrics: FID CLIPScore MSE PSNR SSIM LPIPS inference time sampling steps
Baselines: DDIM Null-Text Inversion Negative Prompt Inversion StyleDiffusion CycleDiffusion Direct Inversion Prompt-to-Prompt MasaCtrl Plug-and-Play
Builds on: Denoising Diffusion Implicit Models Latent Consistency Models Prompt-to-Prompt MasaCtrl Direct Inversion
Closest papers by meaning
- Tuning-Free Inversion-Enhanced Control for Consistent Image Editing (AAAI 2024 · similarity 0.93)
- Invertible Consistency Distillation for Text-Guided Image Editing in Around 7 Steps (NeurIPS 2024 · similarity 0.92)
- DCI: Dual-Conditional Inversion for Boosting Diffusion-Based Image Editing (NeurIPS 2025 · similarity 0.92)
- PnP Inversion: Boosting Diffusion-based Editing with 3 Lines of Code (ICLR 2024 · similarity 0.92)
- High-Fidelity Diffusion-Based Image Editing (AAAI 2024 · similarity 0.92)
- Object-Aware Inversion and Reassembly for Image Editing (ICLR 2024 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.