Research map / Image generation / Editing and style transfer
Prompt Augmentation for Self-supervised Text-guided Image Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes prompt augmentation (BLIP captions expanded via BERT/NLTK into target prompts) plus contrastive and soft contrastive losses on a latent diffusion model to learn self-supervised localised text-guided image editing without paired data or inference-time masks.
Problem
Text-guided image manipulation must both transform image content per the text and preserve context; existing methods either rely on inference-time masks, domain-specific training, or paired data.
Approach
Augment a single source prompt into several target prompts, derive an on-the-fly attention mask from noise-estimate differences across prompts during training, and apply a contrastive loss (plus a similarity-weighted Soft-CL) that pushes edited regions apart and pulls preserved regions together.
Key results
- Full method reaches CLIPScore 78.19%, FID 133, SSIM-M 70.39% versus SDEdit's 76.76, 174, 63.07.
- Ablation: adding CL alone improves CLIPScore to 77.94 and FID to 159; Soft-CL plus Soft-PA gives the best overall results.
- Human study shows preference over SDEdit (76.3%), DiffEdit (73.84%), DALL-E 2 (68.3%), and InstructPix2Pix (62.15%) for the paper's method, per the table's split.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add contrastive loss (CL) (ablation) | Proposed image manipulation method · Ablation evaluation | CLIPScore (%): 76.76 -> 77.94 | Helped |
| Add contrastive loss (CL) (ablation) | Proposed image manipulation method · Ablation evaluation | FID: 174 -> 159 | Helped |
| Add contrastive loss (CL) (ablation) | Proposed image manipulation method · Ablation evaluation | SSIM-M (%): 63.07 -> 68.37 | Helped |
| Replace CL with Soft-CL (ablation) | Proposed image manipulation method · Ablation evaluation | CLIPScore (%): 76.76 -> 77.53 | Helped |
| Replace CL with Soft-CL (ablation) | Proposed image manipulation method · Ablation evaluation | FID: 174 -> 140 | Helped |
| Replace CL with Soft-CL (ablation) | Proposed image manipulation method · Ablation evaluation | SSIM-M (%): 63.07 -> 67.21 | Helped |
| Add Soft-PA to Soft-CL (ablation) | Proposed image manipulation method · Ablation evaluation | CLIPScore (%): 77.53 -> 78.19 | Helped |
| Add Soft-PA to Soft-CL (ablation) | Proposed image manipulation method · Ablation evaluation | FID: 140 -> 133 | Helped |
| Add Soft-PA to Soft-CL (ablation) | Proposed image manipulation method · Ablation evaluation | SSIM-M (%): 67.21 -> 70.39 | Helped |
Surprising findings
- DALL-E 2 achieves the highest CLIPScore and SSIM-M yet is preferred less than the proposed method in the human study, attributed to low fidelity to masked content.
Limitations
- Background preservation (SSIM-M 70.39) lags behind DALL-E 2 and DiffEdit, which are not directly comparable due to inference-time masks.
- Evaluation set is small (135 images) and relies on automatic metrics with limited human study detail.
Experimental setup
Datasets: LAION-5B COCO ImageNet
Benchmarks: COCO ImageNet generated images from Stable Diffusion v1.4 (135-image evaluation set)
Metrics: CLIPScore SSIM-M FID CLIP-R-Precision human preference study
Baselines: SDEdit DiffEdit DALL-E 2 InstructPix2Pix
Builds on: Latent Diffusion Model DiffEdit Prompt-to-Prompt SDEdit
Closest papers by meaning
- Textualize Visual Prompt for Image Editing via Diffusion Bridge (AAAI 2025 · similarity 0.90)
- Text-Driven Image Editing via Learnable Regions (CVPR 2024 · similarity 0.90)
- An Item Is Worth a Prompt: Versatile Image Editing with Disentangled Control (AAAI 2025 · similarity 0.89)
- Preserve or Modify? Context-Aware Evaluation for Balancing Preservation and Modification in Text-Guided Image Editing (CVPR 2025 · similarity 0.88)
- Devil is in the Detail: Towards Injecting Fine Details of Image Prompt in Image Generation via Conflict-free Guidance and Stratified Attention (CVPR 2025 · similarity 0.88)
- Multi-Region Text-Driven Manipulation of Diffusion Imagery (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.