Research map / Image generation / Fast diffusion sampling
Structure Matters: Tackling the Semantic Discrepancy in Diffusion Models for Image Inpainting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes StrDiffusion, a structure-guided diffusion model for image inpainting that uses a progressively sparser auxiliary structure (grayscale to edge) to guide texture denoising and reduce the semantic discrepancy between masked and unmasked regions.
Problem
Diffusion-based image inpainting produces a semantic discrepancy between masked and unmasked regions because dense unmasked texture is not fully degraded during the forward process while masked regions become pure noise.
Approach
Reformulates the texture denoising objective under guidance from an auxiliary structure (grayscale map degrading to masked edge plus noise) whose semantic sparsity increases over time, uses a spatially adaptive normalization network to inject structure guidance, and adds a discriminator-based adaptive resampling strategy that regulates texture-structure correlation.
Key results
- On Places2 with 10-20% irregular masks, StrDiffusion reaches 29.45 PSNR, 0.973 SSIM and 8.75 FID, versus 28.98/0.961/12.03 for IR-SDE.
- On Places2 with 40-50% masks, StrDiffusion achieves FID 36.93 compared with 39.77 for IR-SDE and 41.16 for RePaint*.
- Ablation on PSV: the full adaptive resampling variant (case D) improves hard-mask FID from 55.73 (IR-SDE with resampling baseline A) to 42.28, and removing the discriminator (case B) degrades results by up to 4.16%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use StrDiffusion instead of the strongest listed baseline | Image inpainting on Places2 irregular masks · Places2 irregular masks 10-20% | PSNR: 29.28 -> 29.45 | Helped |
| Use StrDiffusion instead of the strongest listed baseline | Image inpainting on Places2 irregular masks · Places2 irregular masks 20-30% | PSNR: 25.88 -> 26.28 | Helped |
| Use StrDiffusion instead of the strongest listed baseline | Image inpainting on Places2 irregular masks · Places2 irregular masks 30-40% | PSNR: 23.60 -> 24.43 | Helped |
| Use StrDiffusion instead of the strongest listed baseline | Image inpainting on Places2 irregular masks · Places2 irregular masks 40-50% | PSNR: 22.18 -> 22.37 | Helped |
| Use StrDiffusion instead of the strongest listed baseline | Image inpainting on Places2 irregular masks · Places2 irregular masks 10-20% | FID: 11.16 -> 8.750 | Helped |
| Use StrDiffusion instead of the strongest listed baseline | Image inpainting on Places2 irregular masks · Places2 irregular masks 20-30% | FID: 20.62 -> 16.16 | Helped |
| Use StrDiffusion instead of the strongest listed baseline | Image inpainting on Places2 irregular masks · Places2 irregular masks 30-40% | FID: 28.49 -> 23.72 | Helped |
| Use StrDiffusion instead of the strongest listed baseline | Image inpainting on Places2 irregular masks · Places2 irregular masks 40-50% | FID: 39.77 -> 36.93 | Helped |
| Add the discriminator to adaptive resampling (ablation) | StrDiffusion · PSV hard mask | PSNR: 25.47 -> 25.94 | Helped |
| Add the discriminator to adaptive resampling (ablation) | StrDiffusion · PSV easy mask | FID: 28.43 -> 26.74 | Helped |
| Add adaptive resampling (ablation) | StrDiffusion · PSV hard mask | PSNR: 25.59 -> 25.94 | Helped |
| Add adaptive resampling (ablation) | StrDiffusion · PSV easy mask | FID: 28.20 -> 26.74 | Helped |
Surprising findings
- Fixed-sparsity structure guidance (gray2gray or edge2edge) performs worse than progressively sparser guidance, indicating time-dependent sparsity is essential.
Limitations
- Structure guidance depends on the semantic correlation between structure and texture, requiring a discriminator and adaptive resampling to control it.
- Dense-texture IR-SDE and RePaint still show residual margins, indicating discrepancy is reduced but not eliminated in all cases.
- Evaluated only at 256x256 on three datasets.
Experimental setup
Datasets: Paris StreetView CelebA Places2
Benchmarks: Places2 Paris StreetView CelebA
Metrics: PSNR SSIM FID KL divergence
Baselines: RFR PENNet HiFill FAR HAN CMT MEDFE CTSDG ZITS RePaint IR-SDE
Builds on: DDPM IR-SDE RePaint SPADE
Closest papers by meaning
- RAD: Region-Aware Diffusion Models for Image Inpainting (CVPR 2025 · similarity 0.89)
- Paint by Inpaint: Learning to Add Image Objects by Removing Them First (CVPR 2025 · similarity 0.87)
- HD-Painter: High-Resolution and Prompt-Faithful Text-Guided Image Inpainting with Diffusion Models (ICLR 2025 · similarity 0.87)
- PrefPaint: Aligning Image Inpainting Diffusion Model with Human Preference (NeurIPS 2024 · similarity 0.87)
- Image Inpainting via Tractable Steering of Diffusion Models (ICLR 2024 · similarity 0.87)
- InpDiffusion: Image Inpainting Localization via Conditional Diffusion Models (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.