Research map / Image generation / Autoregressive and latent generation
Towards Enhanced Image Inpainting: Mitigating Unwanted Object Insertion and Preserving Color Consistency
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ASUKA, a post-training method that adds an MAE-based context prior and an inpainting-specific VAE decoder to frozen SD 1.5 and FLUX inpainting models to reduce unwanted object insertion and mask-unmask color shift.
Problem
Latent generative inpainting models often hallucinate unwanted objects in masked regions and produce color shifts between inpainted and unmasked areas, and retraining the full three-stage pipeline is costly.
Approach
Freeze the generator and VAE encoder, replace the text condition with an MAE reconstruction prior aligned through a small trainable alignment module, and train a VAE decoder conditioned on the unmasked pixels as a local harmonization task with color and latent augmentation.
Key results
- ASUKA-SD reaches FID 11.495 on MISATO versus 12.812 for SD, with U-IDS 0.423 vs 0.345 and Gradient@edge 47.753 vs 63.844.
- On Places2, ASUKA-SD achieves FID 1.230 versus 1.514 for SD and P-IDS 0.287 versus 0.207.
- ASUKA-FLUX improves FLUX on MISATO@512 with FID 11.372 vs 12.839 and Gradient@edge 48.635 vs 65.928.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add ASUKA alignment and decoder to Stable Diffusion v1.5 | Stable Diffusion v1.5 · MISATO | P-IDS: 0.211 -> 0.312 | Helped |
| Add ASUKA alignment and decoder to Stable Diffusion v1.5 | Stable Diffusion v1.5 · MISATO | Gradient@edge: 63.844 -> 47.753 | Helped |
| Add ASUKA alignment and decoder to Stable Diffusion v1.5 | Stable Diffusion v1.5 · Places 2 | P-IDS: 0.207 -> 0.287 | Helped |
| Add ASUKA alignment and decoder to Stable Diffusion v1.5 | Stable Diffusion v1.5 · Places 2 | Gradient@edge: 160.705 -> 147.733 | Helped |
| Add ASUKA alignment and decoder to FLUX | FLUX · MISATO@512 | P-IDS: 0.223 -> 0.327 | Helped |
| Add ASUKA alignment and decoder to FLUX | FLUX · MISATO@512 | Gradient@edge: 65.928 -> 48.635 | Helped |
| Add a decoder conditioned on unmasked image pixels (ablation) | ASUKA-SD · MISATO | Gradient@edge: 63.142 -> 48.588 | Helped |
| Add color augmentation during decoder training (ablation) | ASUKA-SD · MISATO | Gradient@edge: 63.142 -> 49.538 | Helped |
| Add latent augmentation and the full decoder training procedure (ablation) | ASUKA-SD · MISATO | Gradient@edge: 49.538 -> 47.753 | Helped |
| Use linear feature-dimension alignment only (ablation) | ASUKA-SD · MISATO | P-IDS: 0.253 -> 0.263 | Helped |
| Add one self-attention block after linear alignment (ablation) | ASUKA-SD · MISATO | P-IDS: 0.263 -> 0.268 | Helped |
| Use four self-attention blocks in the alignment module (ablation) | ASUKA-SD · MISATO | P-IDS: 0.268 -> 0.312 | Helped |
Surprising findings
- Pixel-based GAN inpainting models score better on Gradient@edge than all latent SD variants, suggesting color shifts originate in the compressed latent space.
- Simply conditioning the VAE decoder on unmasked pixels (prior work) only partly fixes color inconsistency; explicit color augmentation training is needed.
Limitations
- Frozen generator limits LPIPS improvement; ASUKA-SD gets second-best LPIPS on some benchmarks.
- Evaluated only on SD 1.5 and FLUX inpainting models and two datasets.
Experimental setup
Datasets: Places2 COCO object segments Co-Mod masks LaMa masks Matterport3D Flickr-Landscape MegaDepth COCO 2014
Benchmarks: Places2 MISATO
Metrics: LPIPS FID U-IDS P-IDS CLIP@mask Gradient@edge
Baselines: Co-Mod MAT LaMa MAE-FAR SD-Repaint SD SD-text SD-token SD-IP SD-T2I SD-CAEv2 SD-LaMa
Builds on: Stable Diffusion inpainting FLUX inpainting MAE-FAR Masked Auto-Encoder Co-Mod
Closest papers by meaning
- Paint by Inpaint: Learning to Add Image Objects by Removing Them First (CVPR 2025 · similarity 0.90)
- IMFine: 3D Inpainting via Geometry-guided Multi-view Refinement (CVPR 2025 · similarity 0.88)
- MTADiffusion: Mask Text Alignment Diffusion Model for Object Inpainting (CVPR 2025 · similarity 0.88)
- DesignEdit: Unify Spatial-Aware Image Editing via Training-free Inpainting with a Multi-Layered Latent Diffusion Framework (AAAI 2025 · similarity 0.88)
- Instant3dit: Multiview Inpainting for Fast Editing of 3D Objects (CVPR 2025 · similarity 0.87)
- Keyframe-Guided Creative Video Inpainting (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.