Research map / Image generation / Editing and style transfer
Contrastive Denoising Score for Text-guided Latent Diffusion Image Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adds a patch-wise contrastive (CUT-style) loss computed on latent self-attention features to Delta Denoising Score, enabling training-free, structure-preserving text-guided image editing with Stable Diffusion and NeRF.
Problem
Delta Denoising Score (DDS) edits images with text prompts but fails to preserve structural details of the source image, and prior CUT-based structure preservation requires auxiliary encoders or has not been applied to latent diffusion models.
Approach
Integrates a CUT patchNCE loss into the DDS framework for Stable Diffusion, computing it on intermediate self-attention features of the denoising U-Net (no auxiliary network, no fine-tuning), with random patches at the same spatial locations as positives and other patches as negatives.
Key results
- On cat2dog, CDS reaches 97.5% CLIP accuracy with DINO structure distance 0.020 and LPIPS 0.079, versus DDS at 97.9%, 0.023, 0.080.
- On cat2cow, CDS achieves structure distance 0.033 and LPIPS 0.112, improving over DDS (0.040, 0.116) at 97.9% CLIP accuracy versus 99.6% for DDS.
- In a 20-participant user study, CDS scored highest on text match (4.43), structure (4.65), and quality (4.20) versus DDS (4.06, 4.05, 3.64).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace DDS with CDS by adding self-attention CUT loss | DDS · cat2dog | Dist: 0.023 -> 0.020 | Helped |
| Replace DDS with CDS by adding self-attention CUT loss | DDS · cat2dog | LPIPS: 0.080 -> 0.079 | Helped |
| Replace DDS with CDS by adding self-attention CUT loss | DDS · cat2dog | CLIP Acc: 97.9% -> 97.5% | Hurt |
| Replace DDS with CDS by adding self-attention CUT loss | DDS · cat2cow | Dist: 0.040 -> 0.033 | Helped |
| Replace DDS with CDS by adding self-attention CUT loss | DDS · cat2cow | LPIPS: 0.116 -> 0.112 | Helped |
| Replace DDS with CDS by adding self-attention CUT loss | DDS · cat2cow | CLIP Acc: 99.6% -> 97.9% | Hurt |
| Add self-attention CUT loss to vanilla DDS (ablation) | CDS · image editing | structural consistency | Helped |
| Apply CUT loss to self-attention features rather than score output or cross-attention features (ablation) | CDS · image editing | structural consistency | Helped |
| Vary the contrastive loss weight (ablation) | CDS · image editing | structural consistency | Mixed |
| Add feature injection following the PnP approach to DDS (ablation) | DDS · image editing | structural consistency | Mixed |
| Add self-attention CUT loss to DDS for NeRF editing (ablation) | CDS · NeRF editing | structural consistency | Helped |
Surprising findings
- Applying the CUT loss to cross-attention features or directly to score outputs over-constrains information and fails to preserve structure, so self-attention features were needed.
- On cat2cow, DDS obtains slightly higher CLIP accuracy (99.6%) than CDS (97.9%).
Limitations
- Failure cases arise from unfavorable random patch selection.
- Source objects in unconventional poses (e.g., looking back) can cause failures.
- Evaluation limited to a small set of 250 cat images and a 20-person user study.
Experimental setup
Datasets: LAION 5B (250 cat images)
Benchmarks: cat2dog cat2cow cat2pig
Metrics: CLIP accuracy DINO-ViT structure distance LPIPS user study score (text match, structure, quality)
Baselines: SDEdit + word swap DiffuseIT DDPM inv + P2P DDPM inv + PnP DDS ED-NeRF
Builds on: Delta Denoising Score Score Distillation Sampling Contrastive Unpaired Translation Plug-and-Play Prompt-to-Prompt
Closest papers by meaning
- Denoising Diffusion Bridge Models (ICLR 2024 · similarity 0.90)
- An Edit Friendly DDPM Noise Space: Inversion and Manipulations (CVPR 2024 · similarity 0.89)
- High-Fidelity Diffusion-Based Image Editing (AAAI 2024 · similarity 0.89)
- InstaRevive: One-Step Image Enhancement via Dynamic Score Matching (ICLR 2025 · similarity 0.88)
- One Stone with Two Birds: A Null-Text-Null Frequency-Aware Diffusion Models for Text-Guided Image Inpainting (NeurIPS 2025 · similarity 0.88)
- TiNO-Edit: Timestep and Noise Optimization for Robust Diffusion-Based Image Editing (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.