Research map / Image generation / GANs
Image Inpainting via Iteratively Decoupled Probabilistic Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the pixel spread model (PSM), an iterative inpainting network that predicts a mean (adversarially trained) and variance (Gaussian NLL-regularized) per pixel, keeps low-uncertainty pixels each iteration, and uses uncertainty-guided attention to reach state-of-the-art large-hole inpainting at 512x512 with fewer iterations and parameters than diffusion models.
Problem
Large-hole, high-resolution image inpainting is hard: one-shot GANs struggle with large missing regions, while autoregressive and diffusion models are computationally expensive.
Approach
Decoupled probabilistic modeling: the mean is optimized with adversarial and perceptual losses while the variance is trained only by a Gaussian NLL with stop-gradient on the mean; the variance serves as an uncertainty map to select reliable pixels for iterative spreading and to bias attention weights.
Key results
- On 512x512 Places2 under large masks, PSM achieves FID 1.68 vs. 2.76 for LDM (387M params) and 2.90 for MAT, with 74M parameters.
- PSM runs about 250 ms per 512x512 image, roughly 10x faster than LDM (~3 s).
- Ablations show 3 training iterations and uncertainty-guided attention each improve FID (full model 2.36 vs. 2.44 without UGA on 256x256 Places365).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use iterative generation instead of one training iteration (ablation) | PSM · Places365-Standard 256×256 | FID: 2.95 -> 2.36 | Helped |
| Use 2 rather than 3 training iterations (ablation) | PSM · Places365-Standard 256×256 | FID: 2.36 -> 2.55 | Hurt |
| Add decoupled probabilistic modeling (ablation) | PSM · Places365-Standard 256×256 | FID: 2.49 -> 2.36 | Helped |
| Add continuous sampling (ablation) | PSM · Places365-Standard 256×256 | FID: 2.45 -> 2.36 | Helped |
| Add uncertainty-guided attention (ablation) | PSM · Places365-Standard 256×256 | FID: 2.44 -> 2.36 | Helped |
| Use attention at 32×32 as well as 16×16, rather than only at 16×16 (ablation) | PSM · Places365-Standard 256×256 | FID: 2.64 -> 2.36 | Helped |
| Increase testing iterations from 4 to 9 (ablation) | PSM · Places365-Standard 256×256 | FID: 2.23 -> 2.05 | Helped |
| Increase testing iterations from 9 to 10 (ablation) | PSM · Places365-Standard 256×256 | FID: 2.05 -> 2.05 | No clear effect |
| Use a uniform mask schedule rather than cubic, cosine, or square root (ablation) | PSM · Places365-Standard 256×256 | FID: 2.54, 2.48, 2.47 -> 2.36 | Helped |
| Use PSM instead of LDM | Image inpainting · Places2 512×512, large mask | FID: 2.76 -> 1.68 | Helped |
| Use PSM instead of MAT | Image inpainting · CelebA-HQ 512×512, large mask | FID: 4.86 -> 4.05 | Helped |
| Use PSM instead of MAT | Image inpainting · Places2 1024×1024, large mask | FID: 5.83 -> 3.95 | Helped |
Surprising findings
- Jointly training the variance with the sampled output caused the variance to collapse to near zero, so the stop-gradient decoupling was necessary.
- The authors note LPIPS may be unsuitable for large-hole inpainting because it is computed pixel-by-pixel, so they report it only for reference.
Limitations
- Tends to make changes in small details rather than large structures, limiting diversity.
- Struggles to recover objects when only a few hints are given (failure cases in supplement).
- Lower recall than LDM, indicating less diversity.
Experimental setup
Datasets: Places2 CelebA-HQ Places365-Standard
Benchmarks: Places2 CelebA-HQ
Metrics: FID P-IDS U-IDS LPIPS Precision Recall PSNR SSIM
Baselines: MAT CoModGAN LaMa LDM Stable Diffusion inpainting RePaint ICT MADF AOT GAN HFill DeepFill v2 EdgeConnect
Builds on: denoising diffusion models StyleGAN2 MAT perceptual loss
Closest papers by meaning
- PD-GAN: Probabilistic Diverse GAN for Image Inpainting (CVPR 2021 · similarity 0.91)
- Image Inpainting via Tractable Steering of Diffusion Models (ICLR 2024 · similarity 0.89)
- InpDiffusion: Image Inpainting Localization via Conditional Diffusion Models (AAAI 2025 · similarity 0.87)
- RAD: Region-Aware Diffusion Models for Image Inpainting (CVPR 2025 · similarity 0.87)
- Neuroexplicit Diffusion Models for Inpainting of Optical Flow Fields (ICML 2024 · similarity 0.86)
- TD-Paint: Faster Diffusion Inpainting Through Time-Aware Pixel Conditioning (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.