Research map / Image generation / Editing and style transfer
Unsupervised Region-Based Image Editing of Denoising Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Region-Based Editing (RBE), an unsupervised method that finds semantic directions in h-space of pre-trained DDPM/iDDPM models by projecting the masked-region Jacobian orthogonal to the unmasked-region Jacobian, enabling local edits without training.
Problem
Discovering and controlling local semantic attributes in the latent (h-space) of pre-trained diffusion models without external supervision such as text, attribute labels, or segmentation masks, while keeping non-edited regions unchanged.
Approach
Compute Jacobians of the noise prediction w.r.t. h-space for a masked (edit) region and for the unmasked region, then orthogonally project the masked Jacobian against the unmasked Jacobian so semantic directions change the target region without altering the rest of the image; only a coarse bounding-box mask is needed.
Key results
- On CelebA-HQ smile editing, RBE achieves FID 14.22 and ID 0.8627 versus Semdiff's 21.77 and 0.7224, Asyrp's 54.04 and 0.4892.
- RBE reports LPIPS 0.0289 for smile editing, better than Semdiff (0.0424), Asyrp (0.2172) and Boundary Diffusion (0.1765).
- Authors claim that for some face attributes the unsupervised method surpasses supervised approaches; experiments run on a single RTX 3090.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RBE instead of Semdiff for smile editing | Local diffusion-model image editing · CelebA-HQ, smile editing | FID: 21.77 -> 14.22 | Helped |
| Use RBE instead of Semdiff for smile editing | Local diffusion-model image editing · CelebA-HQ, smile editing | ID: 0.7224 -> 0.8627 | Helped |
| Use RBE instead of Semdiff for smile editing | Local diffusion-model image editing · CelebA-HQ, smile editing | MSE (×10^-4): 5.23 -> 5.74 | Hurt |
| Use RBE instead of Semdiff for smile editing | Local diffusion-model image editing · CelebA-HQ, smile editing | LPIPS: 0.0424 -> 0.0289 | Helped |
| Project the masked-region Jacobian orthogonally to the unmasked-region Jacobian | RBE local image editing · CelebA-HQ, eyes and mouth editing | Unmasked-region pixel change | Helped |
| Use a rectangular bounding box rather than a segmentation mask | RBE semantic discovery · Local image regions | Semantic change richness | Helped |
Surprising findings
- The unsupervised method outperforms supervised Asyrp and Boundary Diffusion on smile editing despite lacking attribute labels.
Limitations
- Small masks (e.g., eyes, teeth) yield no significant semantic directions.
- Global attributes require expanding the mask to large areas, where the method is not clearly better than prior global methods.
- Evaluation limited mainly to CelebA-HQ with 500 test images and a single smile comparison table.
Experimental setup
Datasets: CelebA-HQ LSUN-church LSUN-bedroom
Benchmarks: CelebA-HQ
Metrics: FID ID MSE LPIPS
Baselines: Asyrp Boundary Diffusion SemanticDiff
Builds on: Asyrp Diffusion autoencoders SemanticDiff Boundary Diffusion ReSeFa
Closest papers by meaning
- Exploring Low-Dimensional Subspace in Diffusion Models for Controllable Image Editing (NeurIPS 2024 · similarity 0.93)
- NoiseCLR: A Contrastive Learning Approach for Unsupervised Discovery of Interpretable Directions in Diffusion Models (CVPR 2024 · similarity 0.92)
- CleanDIFT: Diffusion Features without Noise (CVPR 2025 · similarity 0.90)
- Interpretable Diffusion via Information Decomposition (ICLR 2024 · similarity 0.89)
- Masked Autoencoders Are Effective Tokenizers for Diffusion Models (ICML 2025 · similarity 0.89)
- Rethinking the Spatial Inconsistency in Classifier-Free Diffusion Guidance (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.