Research map / Image generation / Autoregressive and latent generation
SmartMask: Context Aware High-Fidelity Mask Generation for Fine-grained Object Insertion and Layout Control
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SmartMask, a context-aware latent diffusion model that predicts fine-grained object masks from semantic layouts (trained on a new amodal segmentation dataset), enabling mask-free object insertion, controllable mask generation, and iterative layout design with an LLaVA-based planner.
Problem
Object insertion and inpainting rely on coarse bounding boxes or scribbles, which hurt background preservation and make high-fidelity masks hard for novice users; generating fine-grained semantic layouts from scratch is also difficult.
Approach
Model object insertion in semantic space: train a diffusion model (initialized from Stable Diffusion v1.5) to predict the next amodal instance mask given an ordered intermediate semantic layout, scene caption, and object label, using large-scale amodal segmentation data; adapt it to user guidance by blending guidance masks into the layout input, and plan multi-object layouts with an instruction-tu
Key results
- SmartMask achieves Norm. L2-BG of 0.098 versus 0.289 for Adobe Gen-Fill and 0.304 for SmartBrush, indicating much better background preservation.
- In a user study on mask quality, SmartMask masks are preferred over SmartBrush mask head 91.67% win vs 5.56% lose.
- SmartMask reports Local-FID 19.21 and CLIP-Score 0.261, comparable to SmartBrush (17.94, 0.262).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SmartMask-generated fine-grained masks instead of coarse-mask inpainting | SmartMask with ControlNet-Inpaint · Image inpainting | Norm. L2-BG: 0.304 -> 0.098 | Helped |
| Use SmartMask-generated fine-grained masks instead of coarse-mask inpainting | SmartMask with ControlNet-Inpaint · Image inpainting | CLIP-Score: 0.262 -> 0.261 | No clear effect |
| Use SmartMask-generated fine-grained masks instead of coarse-mask inpainting | SmartMask with ControlNet-Inpaint · Image inpainting | Local-FID: 17.94 -> 19.21 | Hurt |
| Use SmartMask mask suggestions instead of SD Inpaint + HQSAM | SmartMask · Generated mask quality | Win: 92.04% | No clear effect |
| Use SmartMask mask suggestions instead of Blended L-Diff + HQSAM | SmartMask · Generated mask quality | Win: 88.91% | No clear effect |
| Use SmartMask mask suggestions instead of SmartBrush + HQSAM | SmartMask · Generated mask quality | Win: 63.89% | Helped |
| Use SmartMask mask suggestions instead of SmartBrush Mask | SmartMask · Generated mask quality | Win: 91.67% | Helped |
Surprising findings
- Inpaint + HQSAM can produce scene-unaware masks, e.g., a woman appearing to float because the inpainting tool added an unrequested chair.
Limitations
- Object insertion quality (Local-FID) is not better than SmartBrush or SD-Inpaint, trading some realism for background preservation.
- Relies on a panoptic segmentation model and a ControlNet-Inpaint model for the full pipeline.
- Evaluation of object insertion is limited to a single benchmark and small-scale user study.
Experimental setup
Datasets: COCO-A KINS in-house amodal segmentation dataset (32,785 images)
Benchmarks: Adobe SmartBrush object insertion evaluation (Local-FID, CLIP-Score, Norm. L2-BG)
Metrics: Local-FID CLIP-Score Norm. L2-BG user study win rate
Baselines: SD-Inpaint SDXL-Inpaint Blended Latent Diffusion Adobe SmartBrush Adobe Gen-Fill HQSAM with inpainting
Builds on: Stable Diffusion v1.5 ControlNet LLaVA SmartBrush Blended Latent Diffusion
Closest papers by meaning
- Click2Mask: Local Editing with Dynamic Mask Generation (AAAI 2025 · similarity 0.89)
- OverLayBench: A Benchmark for Layout-to-Image Generation with Dense Overlaps (NeurIPS 2025 · similarity 0.85)
- FreeMask: Rethinking the Importance of Attention Masks for Zero-Shot Video Editing (AAAI 2025 · similarity 0.84)
- Instant3dit: Multiview Inpainting for Fast Editing of 3D Objects (CVPR 2025 · similarity 0.84)
- Zero-Painter: Training-Free Layout Control for Text-to-Image Synthesis (CVPR 2024 · similarity 0.84)
- Towards Enhanced Image Inpainting: Mitigating Unwanted Object Insertion and Preserving Color Consistency (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.