Research map / Image generation / Editing and style transfer
Spin: Diffusion-based Semantic Image Painting Through Independent Information Injection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Spin, a diffusion-based semantic image painting framework that uses a saliency focus loss for textual-inversion embeddings, attention-based region guidance for mask-confined semantic injection, and style guidance to align masked and background style.
Problem
Semantic image painting (filling masked regions conditioned on a text or reference image) suffers from semantic inconsistency, unnatural boundary transitions, and style inconsistency with the background, especially with image conditions.
Approach
Decouple salient from non-salient reference content via a saliency-focused textual inversion loss, confine semantic cross-attention to the masked region through attention-based region guidance (keeping original self-attention for background information), and steer denoising with a pixel-space style loss computed against the background image.
Key results
- Image-conditioned SIP: LPIPS_BG 0.06, CLIP-I 83.70, CLIP-T 28.80, user score 7.27, outperforming TF (CLIP-I 82.86) and others.
- Text-conditioned SIP: CLIP-T 29.13 and NIMA 5.03, best among Blended, BLD, SD-inpaint, PPt, Uni.
- Ablation: adding attention-based region guidance raises CLIP-I from 73.80 (saliency focus alone) to 82.66.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Spin instead of the strongest listed baseline, Blended | Text-conditioned SIP · Text-conditioned SIP | CLIPT: 28.87 -> 29.13 | Helped |
| Use Spin instead of Uni | Text-conditioned SIP · Text-conditioned SIP | NIAM: 4.88 -> 5.03 | Helped |
| Use Spin instead of Uni | Text-conditioned SIP · Text-conditioned SIP | LPIPSBG: 0.08 -> 0.06 | No clear effect |
| Use Spin instead of TF | Image-conditioned SIP · Image-conditioned SIP | CLIPI: 82.86 -> 83.70 | Helped |
| Use Spin instead of TF | Image-conditioned SIP · Image-conditioned SIP | CLIPT: 28.11 -> 28.80 | Helped |
| Use Spin instead of TF | Image-conditioned SIP · Image-conditioned SIP | SUser: 7.18 -> 7.27 | Helped |
| Add background preservation to saliency-focused embedding extraction (ablation) | Spin · Image-conditioned SIP | LPIPSBG: 0.63 -> 0.08 | Helped |
| Add Attention-based Region Guidance (ARG) to the background-preserved variant (ablation) | Spin · Image-conditioned SIP | CLIPI: 77.14 -> 82.66 | Helped |
| Add ARG to the background-preserved variant (ablation) | Spin · Image-conditioned SIP | NIMA: 4.79 -> 5.03 | Helped |
| Add style guidance to saliency-focused embedding extraction (ablation) | Spin · Image-conditioned SIP | NIMA: 4.87 -> 4.88 | No clear effect |
| Combine background preservation, ARG, and style guidance (ablation) | Spin · Image-conditioned SIP | NIMA: 5.03 -> 5.07 | Helped |
Limitations
- Relies on Stable Diffusion v1.4 for most experiments; generality shown only qualitatively on PixArt and NeTI.
- Style guidance is computed with a simple pixel-space style loss and is described as less efficient than other guidance approaches; requires per-reference embedding optimization.
- Quantitative evaluation uses only the real-domain subset, while non-real-domain results are qualitative.
Experimental setup
Datasets: TF dataset (332 sets, four domains)
Benchmarks: TF dataset real-domain test set Internet-collected non-real-domain images
Metrics: LPIPS_BG CLIP-I CLIP-T NIMA User Score
Baselines: Textual Inversion Multi-token TI DreamArtist VCT Blended Diffusion BLD SD-inpaint PPt Uni DIB Any Paint
Builds on: textual inversion DDIM inversion Stable Diffusion RePaint
Closest papers by meaning
- Paint by Inpaint: Learning to Add Image Objects by Removing Them First (CVPR 2025 · similarity 0.86)
- Structure Matters: Tackling the Semantic Discrepancy in Diffusion Models for Image Inpainting (CVPR 2024 · similarity 0.86)
- HD-Painter: High-Resolution and Prompt-Faithful Text-Guided Image Inpainting with Diffusion Models (ICLR 2025 · similarity 0.85)
- Image Inpainting via Tractable Steering of Diffusion Models (ICLR 2024 · similarity 0.85)
- Exploring Low-Dimensional Subspace in Diffusion Models for Controllable Image Editing (NeurIPS 2024 · similarity 0.85)
- NoiseCLR: A Contrastive Learning Approach for Unsupervised Discovery of Interpretable Directions in Diffusion Models (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.