Research map / Image generation / Editing and style transfer
VersaFusion: A Versatile Diffusion-Based Framework for Fine-Grained Image Editing and Enhancement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VersaFusion, a tuning-free dual-branch diffusion editing framework (Guidance and Generation branches on Stable Diffusion v1.5) with proximal negative-prompt inversion and image prompts for fine-grained drag-style editing and related tasks.
Problem
Fine-grained, pixel-level image editing and enhancement with pre-trained text-to-image diffusion models, where existing text-guided and drag-based methods lack precise control and fidelity.
Approach
A dual-branch architecture that reconstructs the source image in a guidance branch and injects its keys/values, features, and gradient guidance into a generation branch, combined with a proximal negative-prompt inversion (PNI) and a CLIP+QFormer image prompt encoder trained on top of frozen SD v1.5.
Key results
- On face manipulation, VersaFusion reports FID 32.98 versus 34.20 for DragonDiffusion and 39.38 for DragGAN (Table 1).
- Its landmark MSE is 11.45, compared with 16.90 for DragonDiffusion and 16.99 for DragDiffusion.
- Inference takes 20.82s for a single point drag, versus 22.50s for DragonDiffusion in the table.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace DragonDiff with VersaFusion | Face manipulation · Face manipulation | landmark distance: 16.90 -> 11.45 | Helped |
| Replace DragonDiff with VersaFusion | Face manipulation · Face manipulation | FID: 34.20 -> 32.98 | Helped |
| Replace DragonDiff with VersaFusion | Face manipulation · Face manipulation | inference time (s): 22.50s -> 20.82s | Helped |
| Replace DragonDiff with VersaFusion | Face manipulation · Face manipulation | preparation time (s): 5.23s -> 5.23s | No clear effect |
| Use PNI inversion instead of DDIM inversion (ablation) | VersaFusion · Image editing | editing quality | Helped |
| Add null-text optimization to DDIM inversion (ablation) | VersaFusion · Image editing | editing accuracy | Helped |
| Add null-text optimization to DDIM inversion (ablation) | VersaFusion · Image editing | time complexity | Hurt |
| Use PNI inversion (ablation) | VersaFusion · Image editing | time complexity: 12 seconds | Helped |
| Remove the image prompt (ablation) | VersaFusion image dragging · Image dragging | visual coherence | Hurt |
| Remove the text prompt (ablation) | VersaFusion image dragging · Image dragging | semantic alignment | Hurt |
| Remove both the image prompt and text prompt (ablation) | VersaFusion image dragging · Image dragging | visual coherence | Hurt |
Surprising findings
- DragGAN achieves the best landmark MSE on aligned faces but does not generalize to general face images, according to the authors.
Limitations
- Editing scenarios requiring significant content imagination remain challenging, per the authors.
- Evaluation is limited to a single face-manipulation table and qualitative figures; no user study.
Experimental setup
Datasets: LAION
Benchmarks: face manipulation landmark test (68 points)
Metrics: landmark MSE distance FID inference time preparation time
Baselines: DragGAN DragDiffusion DragonDiffusion
Builds on: DragonDiffusion DragDiffusion Null-text inversion IP-Adapter Proximal guidance
Closest papers by meaning
- TiNO-Edit: Timestep and Noise Optimization for Robust Diffusion-Based Image Editing (CVPR 2024 · similarity 0.88)
- Inversion-Free Image Editing with Language-Guided Diffusion Models (CVPR 2024 · similarity 0.88)
- Towards Efficient Diffusion-Based Image Editing with Instant Attention Masks (AAAI 2024 · similarity 0.88)
- BARET: Balanced Attention Based Real Image Editing Driven by Target-Text Inversion (AAAI 2024 · similarity 0.87)
- High-Fidelity Diffusion-Based Image Editing (AAAI 2024 · similarity 0.87)
- Tuning-Free Inversion-Enhanced Control for Consistent Image Editing (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.