Research map / Image generation / Autoregressive and latent generation
Don't Look into the Dark: Latent Codes for Pluralistic Image Inpainting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts generative transformers over discrete VQGAN latent codes to pluralistic image inpainting, using a restrictive partial encoder, a bidirectional transformer over visible tokens, and a coupled decoder.
Problem
Pluralistic image inpainting under large masks: existing methods either produce deterministic results, have limited diversity, or lose visual coherence when the missing region is large.
Approach
Encode only visible/near-visible regions into discrete tokens with a restrictive partial convolution (mask threshold alpha=0.5), predict the missing tokens with a BERT-style bidirectional transformer using MaskGIT-style parallel decoding with annealed temperature, and decode by fusing the token map with partial-convolution image priors.
Key results
- On Places with a 80% box mask, FID 13.30 vs. 17.5 for MAT and 19.48 for LaMa; diversity 0.29 on Places box setting, comparable to MaskGIT's 0.39.
- On CelebA-HQ small-mask FID 2.70 vs. 2.94 for MAT; box-mask FID 12.79 vs. 15.18 for MAT.
- Ablation: alpha=0.5 and temperature t=1 with annealing give best FID (12.81 full model); vanilla decoder raises FID to 18.50.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use restrictive encoder and partial-convolution decoder instead of the vanilla network designs (ablation) | Ours · Places Box (3000-image subset) | FID: 15.32; 18.50 -> 12.81 | Helped |
| set sampling temperature to t = 0.1 instead of t = 1.0 (ablation) | Ours · Places Box (3000-image subset) | FID: 12.81 -> 14.62 | Hurt |
| set sampling temperature to t = 0.5 instead of t = 1.0 (ablation) | Ours · Places Box (3000-image subset) | FID: 12.81 -> 13.61 | Hurt |
| set sampling temperature to t = 2 instead of t = 1.0 (ablation) | Ours · Places Box (3000-image subset) | FID: 12.81 -> 13.96 | Hurt |
| set restrictive-convolution α = 0.25 instead of α = 0.5 (ablation) | Ours · Places Box (3000-image subset) | FID: 12.81 -> 12.9 | Hurt |
| set restrictive-convolution α = 0.75 instead of α = 0.5 (ablation) | Ours · Places Box (3000-image subset) | FID: 12.81 -> 14.12 | Hurt |
| set restrictive-convolution α = 1.0 instead of α = 0.5 (ablation) | Ours · Places Box (3000-image subset) | FID: 12.81 -> 15.04 | Hurt |
| replace restrictive partial convolutions with regular convolutions in the encoder (ablation) | Ours · Places Box (3000-image subset) | FID: 12.81 -> 15.32 | Hurt |
| directly decode predicted latent codes instead of coupling them with partial-image priors (ablation) | Ours · Places Box (3000-image subset) | FID: 12.81 -> 18.50 | Hurt |
| use Ours instead of MAT | image inpainting · Places and CelebA-HQ, small and large masks and box | FID: Places: 1.19 -> 1.02; 3.32 -> 2.82; 17.5 -> 13.30; CelebA-HQ: 2.94 -> 2.70; 5.16 -> 5.04; 15.18 -> 12.79 | Helped |
| use Ours instead of MAT | image inpainting · CelebA-HQ Box | Diversity: 0.10 ± 0.02 -> 0.28±0.05 | Helped |
| use Ours instead of MaskGIT | image inpainting · Places Box | Diversity: 0.39±0.05 -> 0.29±0.06 | Hurt |
Surprising findings
- MaskGIT reaches higher diversity on Places than the proposed method, but at a large cost in visual quality (FID 36-53 on Places).
Limitations
- Struggles to complete semantically salient objects such as people, animals, shops and furniture when trained on Places.
- Slower inference than end-to-end single-pass inpainting methods due to iterative sampling (sampling steps k=5).
- Resolution extension and semantic-label training left unexplored.
Experimental setup
Datasets: Places365-Standard CelebA-HQ
Benchmarks: Places365-Standard CelebA-HQ
Metrics: FID LPIPS-based diversity
Baselines: MAT LaMa MaskGIT Pluralistic
Builds on: VQGAN MaskGIT BERT partial convolution Pluralistic inpainting
Closest papers by meaning
- BiGR: Harnessing Binary Latent Codes for Image Generation and Improved Visual Representation Capabilities (ICLR 2025 · similarity 0.87)
- DLFormer: Discrete Latent Transformer for Video Inpainting (CVPR 2022 · similarity 0.87)
- Towards Enhanced Image Inpainting: Mitigating Unwanted Object Insertion and Preserving Color Consistency (CVPR 2025 · similarity 0.86)
- Generative Latent Coding for Ultra-Low Bitrate Image Compression (CVPR 2024 · similarity 0.85)
- DesignEdit: Unify Spatial-Aware Image Editing via Training-free Inpainting with a Multi-Layered Latent Diffusion Framework (AAAI 2025 · similarity 0.85)
- Image Inpainting via Tractable Steering of Diffusion Models (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.