Research map / Image generation / Editing and style transfer
SwiftEdit: Lightning Fast Text-Guided Image Editing via One-Step Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SwiftEdit, a one-step text-guided image editing method that inverts images with a one-step inversion network built on SwiftBrushv2 and edits via attention rescaling with a self-guided mask, running in 0.23 s.
Problem
Text-guided image editing with diffusion models requires costly multi-step inversion and sampling, too slow for real-time and on-device use.
Approach
A one-step inversion network initialized from the one-step generator SwiftBrushv2, trained in two stages (synthetic regression plus real-image DISTS with an SDS-inspired regularizer), together with an image-conditioned IP-Adapter branch and Attention Rescaling for Mask-aware editing (ARaM) that derives editing masks from the difference of inverted noises under source and edit prompts.
Key results
- SwiftEdit edits in 0.23 s on a single A100, at least 50x faster than multi-step methods and about 5x faster than few-step methods (ICD 1.62 s, TurboEdit 1.32 s).
- On PieBench SwiftEdit reaches PSNR 23.33, MSE 6.60e-4, CLIP-Whole 25.16 and CLIP-Edited 21.25, competitive with multi-step and few-step baselines.
- User study with 140 participants: 47.8% preferred SwiftEdit for editing semantics and 40% for background preservation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SwiftEdit instead of multi-step or few-step editing methods | Text-guided image editing · PieBench | Runtime (seconds): 0.23 | Helped |
| Add stage 1 synthetic-image training (ablation) | SwiftEdit inversion framework · Real image reconstruction | PSNR: 22.26 -> 24.35 | Helped |
| Add stage 2 real-image training (ablation) | SwiftEdit inversion framework · Real image reconstruction | PSNR: 17.95 -> 24.35 | Helped |
| Add IP-Adapter (ablation) | SwiftEdit inversion framework · Real image reconstruction | PSNR: 18.57 -> 24.35 | Helped |
| Add stage 1 regression loss (ablation) | SwiftEdit inversion framework · PieBench | CLIP-Whole: 22.98 -> 25.16 | Helped |
| Add stage 2 regularization loss (ablation) | SwiftEdit inversion framework · PieBench | CLIP-Whole: 24.19 -> 25.16 | Helped |
| Add both stage 1 regression and stage 2 regularization losses (ablation) | SwiftEdit inversion framework · PieBench | CLIP-Edited: 19.07 -> 21.25 | Helped |
| Use ground-truth masks instead of self-guided masks (ablation) | SwiftEdit · PieBench | PSNR: 23.33 -> 23.31 | No clear effect |
Surprising findings
- Self-guided masks from the inversion network are nearly as good as ground-truth masks (CLIP-Whole 25.16 vs 25.56 with GT masks, PSNR 23.33 vs 23.31).
Limitations
- Performance depends on the quality of the SBv2 generator, so training data biases transfer to the inversion network.
- Achieves instant (0.23 s) rather than real-time editing.
- PSNR is slightly below NT-Inv + P2P and CLIP-Whole is below TurboEdit.
Experimental setup
Datasets: PieBench JourneyDB CommonCanvas
Benchmarks: PieBench
Metrics: PSNR MSE CLIP-Whole CLIP-Edited runtime SSIM LPIPS
Baselines: DDIM + P2P Null-text Inversion + P2P Direct Inversion + MasaCtrl DDIM + P2P-Zero Direct Inversion + PnP ReNoise TurboEdit ICD
Builds on: SwiftBrushv2 IP-Adapter Direct Inversion Score Distillation Sampling DISTS
Closest papers by meaning
- Object-Aware Inversion and Reassembly for Image Editing (ICLR 2024 · similarity 0.89)
- LEDITS++: Limitless Image Editing using Text-to-Image Models (CVPR 2024 · similarity 0.88)
- Towards Efficient Diffusion-Based Image Editing with Instant Attention Masks (AAAI 2024 · similarity 0.88)
- Inversion-Free Image Editing with Language-Guided Diffusion Models (CVPR 2024 · similarity 0.88)
- BARET: Balanced Attention Based Real Image Editing Driven by Target-Text Inversion (AAAI 2024 · similarity 0.88)
- PnP Inversion: Boosting Diffusion-based Editing with 3 Lines of Code (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.