Research map / Image generation / Text-to-image diffusion
Textualize Visual Prompt for Image Editing via Diffusion Bridge
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Textualizes a before-after image pair into text embeddings using a diffusion bridge built on a single pretrained text-to-image model (SD v1.5), with differential attention control during optimization, enabling visual-prompt image editing without triplet data or retraining.
Problem
Visual prompts (before-after image pairs) convey transformations hard to describe in language, but existing methods need text-guided image-to-image models retrained on triplet data or inversion methods that lose image detail, limiting scalability and generalization of editing.
Approach
Build a deterministic DDIM-based diffusion bridge from one T2I model (null text to invert the before-image, text condition to regenerate the after-image), optimize a text embedding against fixed start and end latents using predicted x0 with a time-aware loss scaling, and inject before-image cross-attention during training (differentiable, multiplication and addition only) to disentangle the transf
Key results
- Ours achieves PSNR 24.57 and LPIPS 0.1197 versus 17.78 and 0.4279 for the best baseline DIA, on the reported real-image evaluation.
- User study overall score 4.72 for Ours versus 3.57 for InstructPix2Pix (highest among compared methods).
- Ablation: removing differential attention drops PSNR to 3.21 and I-CLIP to 0.3213; removing time-aware scaling drops PSNR to 17.23.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use TextVDB instead of the strongest baseline | SD v1.5 image editing · Real images | PSNR: 17.78 -> 24.57 | Helped |
| use TextVDB instead of the strongest baseline | SD v1.5 image editing · Real images | SSIM: 0.5977 -> 0.8091 | Helped |
| use TextVDB instead of the strongest baseline | SD v1.5 image editing · Real images | LPIPS: 0.3168 -> 0.1197 | Helped |
| use TextVDB instead of the strongest baseline | SD v1.5 image editing · Real images | V-CLIP: 0.2674 -> 0.2750 | Helped |
| use TextVDB instead of the strongest baseline | SD v1.5 image editing · Real images | I-CLIP: 0.8114 -> 0.8178 | Helped |
| use TextVDB instead of the strongest baseline | SD v1.5 image editing · Real images | V-DINO: 0.2247 -> 0.3234 | Helped |
| use TextVDB instead of the strongest baseline | SD v1.5 image editing · Real images | I-DINO: 0.8763 -> 0.9073 | Helped |
| use TextVDB instead of the strongest baseline | SD v1.5 image editing · Real images | V-VIE: 3.01 -> 4.48 | Helped |
| use TextVDB instead of the strongest baseline | SD v1.5 image editing · Real images | I-VIE: 4.01 -> 4.29 | Helped |
| use TextVDB instead of the strongest baseline | SD v1.5 image editing · Real images | Edit Analogy: 4.21 -> 4.81 | Helped |
| use TextVDB instead of the strongest baseline | SD v1.5 image editing · Real images | Fidelity: 3.87 -> 4.62 | Helped |
| use TextVDB instead of the strongest baseline | SD v1.5 image editing · Real images | Overall: 3.57 -> 4.72 | Helped |
Surprising findings
- Textual inversion-style optimization is argued to be invalid under deterministic DDIM bridges, since stochastic noise breaks the fixed latent constraint.
- Without time-aware loss scaling, the predicted x0 approaches the target early and then deviates, so the final output is not closest to the target.
Limitations
- Editing real images is limited by the T2I model's prior; many transformations are out of distribution.
- The attention injection timestep must be chosen per editing type, requiring tuning.
- Relies on a coarse descriptor to initialize and size the text embedding.
Experimental setup
Datasets: PIE DreamBooth P2P
Benchmarks: PIE DreamBooth P2P-generated pairs
Metrics: PSNR SSIM LPIPS V-CLIP I-CLIP V-DINO I-DINO V-VIE I-VIE user study edit analogy
Baselines: VII InstructPix2Pix DiffuseIT DIA Analogist
Builds on: DDIB textual inversion classifier-free guidance DDIM Visual Instruction Inversion
Closest papers by meaning
- Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.91)
- RelationAdapter: Learning and Transferring Visual Relation with Diffusion Transformers (NeurIPS 2025 · similarity 0.91)
- Prompting Hard or Hardly Prompting: Prompt Inversion for Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.90)
- Prompt Augmentation for Self-supervised Text-guided Image Manipulation (CVPR 2024 · similarity 0.90)
- AP-Adapter: Improving Generalization of Automatic Prompts on Unseen Text-to-Image Diffusion Models (NeurIPS 2024 · similarity 0.90)
- DreamDistribution: Learning Prompt Distribution for Diverse In-distribution Generation (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.