Research map / Image generation / Editing and style transfer
A Diffusion-Based Framework for Occluded Object Movement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiffOOM, a dual-branch Stable Diffusion framework that de-occludes and moves occluded objects in real images using color-fill initialization, attention-guided masks, LoRA, latent optimization, and local text guidance.
Problem
Moving objects within real-world images when they are partially occluded, which requires completing the hidden part of the object before relocating it and inpainting the original location.
Approach
Decouples occluded object movement into a de-occlusion branch (color-fill initialization, latent hold, cross/self-attention-refined object mask guidance, and LoRA on visible pixels) and a movement branch (noise-filled original region, self-attention key/value injection from inversion, latent optimization toward the target box with pixel-space latent resizing, and local text-guided denoising), run
Key results
- De-occlusion KID 0.0142 vs PCNet 0.0186, and CLIP-T 30.49 vs 29.57.
- Occluded object movement: DINO-OP 0.561 (lower is better, best) and DINO-TP 0.742 and CLIP-TP 0.960, best among compared methods.
- User study with 60 volunteers: winning rates of 75.8%-86.8% over AnyDoor, SD Inpainting, DragDiffusion, DiffEditor.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DiffOOM instead of PCNet for de-occlusion | COCOA de-occlusion · COCOA de-occlusion | KID: 0.0186 -> 0.0142 | Helped |
| use DiffOOM instead of PCNet for de-occlusion | COCOA de-occlusion · COCOA de-occlusion | CLIP-T: 29.57 -> 30.49 | Helped |
| use DiffOOM instead of PBE | COCOA occluded object movement · COCOA occluded object movement | DINO-OP: 0.575 -> 0.561 | Helped |
| use DiffOOM instead of SD Inpainting | COCOA occluded object movement · COCOA occluded object movement | DINO-TP: 0.742 -> 0.742 | No clear effect |
| use DiffOOM instead of DiffEditor | COCOA occluded object movement · COCOA occluded object movement | CLIP-TP: 0.958 -> 0.960 | Helped |
| add Color Fill Strategy (ablation) | DiffOOM · COCOA occluded object movement | DINO-TP: 0.717 -> 0.742 | Helped |
| add Attention Guidance (ablation) | DiffOOM · COCOA occluded object movement | DINO-TP: 0.731 -> 0.742 | Helped |
| add Attention Guidance (ablation) | DiffOOM · COCOA occluded object movement | DINO-OP: 0.612 -> 0.561 | Helped |
| add LoRA (ablation) | DiffOOM · COCOA occluded object movement | DINO-TP: 0.733 -> 0.742 | Helped |
| add Latent Resizing (ablation) | DiffOOM · COCOA occluded object movement | DINO-TP: 0.695 -> 0.742 | Helped |
| add Local Text Guidance (ablation) | DiffOOM · COCOA occluded object movement | CLIP-TP: 0.955 -> 0.960 | Helped |
Surprising findings
- SD Inpainting gets the top DINO-TP score (tied with the method) but the worst DINO-OP, because it leaves residual or regenerated objects at the original location.
Limitations
- Evaluation is limited to a filtered COCOA subset of 120 images and only compares against a few baselines.
- Relies on per-object LoRA fine-tuning and multiple diffusion branches, which adds per-edit cost (not quantified).
Experimental setup
Datasets: COCOA COCO
Benchmarks: COCOA-derived evaluation set (120 images, 150 objects, 1200 cases)
Metrics: KID CLIP-T DINO-OP DINO-TP CLIP-TP user study win rate
Baselines: PCNet Paint-By-Example AnyDoor SD Inpainting DragDiffusion DragonDiffusion DiffEditor MasaCtrl
Builds on: Stable Diffusion v1.5 DragonDiffusion DragDiffusion DiffEditor DDIM inversion
Closest papers by meaning
- Move Anything with Layered Scene Diffusion (CVPR 2024 · similarity 0.89)
- Motion Guidance: Diffusion-Based Image Editing with Differentiable Motion Estimators (ICLR 2024 · similarity 0.88)
- DragonDiffusion: Enabling Drag-style Manipulation on Diffusion Models (ICLR 2024 · similarity 0.88)
- DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editing (CVPR 2024 · similarity 0.88)
- DesignEdit: Unify Spatial-Aware Image Editing via Training-free Inpainting with a Multi-Layered Latent Diffusion Framework (AAAI 2025 · similarity 0.87)
- Diffusion Handles Enabling 3D Edits for Diffusion Models by Lifting Activations to 3D (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.