Research map / Detection and segmentation / Open-set and few-shot detection
Completing Visual Objects via Bridging Generation and Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MaskComp, which alternates a mask-conditioned generation stage (CompNet, a ControlNet variant on frozen Stable Diffusion) with a SAM-based segmentation and mask-voting stage to iteratively denoise a partial object mask for object completion.
Problem
Object completion: reconstructing the full appearance of a partially occluded object from its visible part, where the complete object mask is unavailable and generated shapes often misalign with the visible part.
Approach
Treats the partial mask as a noisy complete mask and runs iterative mask denoising (IMD): a mask-conditioned generator (CompNet) samples several completed images, a segmenter extracts their masks, and thresholded voting over these masks yields a refined mask for the next step, with an auxiliary mask-prediction loss and time-variant gating of the visible-part token.
Key results
- On AHP, MaskComp reaches FID-G 16.9 versus 30.8 for Stable Diffusion 2.1 and 40.2 for ControlNet.
- On DYCE, MaskComp reaches FID-G 20.0 and an average user-study rank of 1.9 versus 3.0 for Stable Diffusion 2.1.
- Ablation shows that conditioning on the complete mask gives FID 12.7 versus 16.9 with the partial mask, and that 5 IMD steps are a good tradeoff.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MaskComp instead of Stable Diffusion 2.1 | Object completion · AHP | FID-G: 30.8 -> 16.9 | Helped |
| Use MaskComp instead of Stable Diffusion 2.1 | Object completion · DYCE | FID-G: 30.0 -> 20.0 | Helped |
| Use SAM instead of CLIPSeg for segmentation (ablation) | MaskComp · AHP | FID: 19.9 -> 16.9 | Helped |
| Increase IMD steps from 1 to 5 (ablation) | MaskComp · AHP | FID: 24.7 -> 16.9 | Helped |
| Increase sampled images from 4 to 6 (ablation) | MaskComp segmentation stage · AHP | FID: 17.4 -> 16.8 | Helped |
| Remove time-variant condition gating (ablation) | MaskComp CompNet · AHP | FID: 16.9 -> 18.2 | Hurt |
| Condition on a more complete mask (ablation) | MaskComp · AHP | FID: 16.9 -> 12.7 | Helped |
| Increase occlusion from 20% to 80% (ablation) | MaskComp · AHP | FID: 13.4 -> 29.9 | Hurt |
| Train without complete objects (ablation) | MaskComp · AHP | FID: 16.9 -> 19.4 | Hurt |
| Remove auxiliary mask prediction supervision (ablation) | MaskComp CompNet · AHP | FID: 16.9 -> 17.7 | Hurt |
| Use logits voting instead of mask voting (ablation) | MaskComp segmentation stage · AHP | FID: 17.6 -> 16.9 | Helped |
| Use more diffusion iterations (ablation) | MaskComp · AHP | FID: 16.9 -> 15.1 | Helped |
Surprising findings
- Averaging generated object masks yields a more realistic shape even though the average of generated images is unrealistic, which the authors use as the basis for mask voting.
- Training on OpenImages without any ground-truth complete objects gives performance only slightly worse than training on AHP with complete objects.
Limitations
- Inference is slow: each IMD step runs multiple diffusion sampling passes, about 15.5 seconds per completion on a V100.
- Generated background objects can appear despite a black background during training.
- Fails on unusual poses that are underrepresented in the AHP training set.
Experimental setup
Datasets: AHP DYCE OpenImages v6
Benchmarks: AHP DYCE
Metrics: FID-G FID-S user study average rank best-ranked percentage
Baselines: ControlNet Kandinsky 2.1 Stable Diffusion 1.5 Stable Diffusion 2.1 amodal segmentation baseline with ControlNet CLIPSeg SEEM
Builds on: ControlNet Stable Diffusion Segment Anything
Closest papers by meaning
- Using Diffusion Priors for Video Amodal Segmentation (CVPR 2025 · similarity 0.86)
- Amodal Ground Truth and Completion in the Wild (CVPR 2024 · similarity 0.85)
- GeoComplete: Geometry-Aware Diffusion for Reference-Driven Image Completion (NeurIPS 2025 · similarity 0.84)
- Any3DIS: Class-Agnostic 3D Instance Segmentation by 2D Mask Tracking (CVPR 2025 · similarity 0.84)
- PartGen: Part-level 3D Generation and Reconstruction with Multi-view Diffusion Models (CVPR 2025 · similarity 0.84)
- Amodal Scene Analysis via Holistic Occlusion Relation Inference and Generative Mask Completion (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.