Research map / Image generation / Fast diffusion sampling
Image Inpainting via Tractable Steering of Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Tiramisu, which steers a pretrained diffusion model's denoising during inpainting by geometric-mean combination with a latent-space probabilistic circuit that exactly computes the constrained posterior.
Problem
Constrained image generation such as inpainting with diffusion models requires the posterior conditioned on known pixels, which is intractable; existing approximations introduce bias.
Approach
Use a scaled-up probabilistic circuit (PC) trained on VQ-GAN latents to exactly compute the constrained posterior under independent soft-evidence constraints in linear time (forward/backward pass), and combine its distribution with the diffusion model's via weighted geometric mean at each denoising step.
Key results
- Tiramisu achieves the lowest average LPIPS (0.421) across 21 dataset-mask settings versus 0.427 for CoPaint and 0.459 for RePaint.
- Tiramisu beats CoPaint in 18 of 21 settings and is best among all methods in 14 of 21 settings.
- Using the PC in about 20% of denoising steps adds roughly 10 s overhead, about 10% of total runtime.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add probabilistic-circuit guidance to CoPaint | Tiramisu · CelebA-HQ, ImageNet, and LSUN-Bedroom; average across 21 dataset-mask settings | LPIPS: 0.427 -> 0.421 | Helped |
| Add probabilistic-circuit guidance to CoPaint | Tiramisu · CelebA-HQ, ImageNet, and LSUN-Bedroom; 21 dataset-mask settings | LPIPS | Helped |
| Add probabilistic-circuit guidance to CoPaint | Tiramisu · CelebA-HQ, ImageNet, and LSUN-Bedroom; 21 dataset-mask settings | LPIPS | Mixed |
| Use PC guidance in a moderate number of denoising steps (around 20%) | Tiramisu · CelebA-HQ Expand1 | LPIPS | Helped |
| Use PC guidance in 20% of denoising steps | Tiramisu · CelebA-HQ Expand1 | additional computational overhead: around 10s; 10% | Hurt |
Surprising findings
- LPIPS first decreases then increases as PC guidance is applied to more denoising steps, so moderate use (around 20%) is best.
Limitations
- Operates in VQ-GAN latent space, so per-pixel tractability is lost and the decoder approximates pixel-space posterior via Monte Carlo.
- Evaluation limited to 100 images per dataset-mask setting and uses only LPIPS plus a small user study.
- Semantic fusion is only preliminary and the paper does not claim it generalizes to other controlled generation tasks.
Experimental setup
Datasets: CelebA-HQ ImageNet LSUN-Bedroom
Benchmarks: CelebA-HQ ImageNet LSUN-Bedroom
Metrics: LPIPS user preference vote difference
Baselines: CoPaint RePaint DDNM DDRM DPS Resampling
Builds on: CoPaint Probabilistic Circuits VQ-GAN latent variable distillation
Closest papers by meaning
- RAD: Region-Aware Diffusion Models for Image Inpainting (CVPR 2025 · similarity 0.90)
- Image Inpainting via Iteratively Decoupled Probabilistic Modeling (ICLR 2024 · similarity 0.89)
- TD-Paint: Faster Diffusion Inpainting Through Time-Aware Pixel Conditioning (ICLR 2025 · similarity 0.89)
- A Simple Approach to Unifying Diffusion-based Conditional Generation (ICLR 2025 · similarity 0.89)
- Fast constrained sampling in pre-trained diffusion models (NeurIPS 2025 · similarity 0.88)
- Rethinking Peculiar Images by Diffusion Models: Revealing Local Minima’s Role (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.