Research map / Detection and segmentation / CNN features
Shadow-Enlightened Image Outpainting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an end-to-end shadow-aware image outpainting framework that extracts instance-level shadows of unobserved objects and uses them to predict scene layouts and align outpainted content with shadows, implemented as a plug-in for SGT.
Problem
Image outpainting typically treats unobserved regions as unknown and ignores the hidden information (shapes, semantics) encoded in shadows cast by unobserved objects, leading to misaligned extensions in complex scenes.
Approach
Extract instance-level shadow regions cast by unobserved areas (with shadow removal to purify semantics, merging/splitting for disjoint/overlapping shadows, and illumination cues), feed these shadow features into scene-layout expansion (GTwE) and layout-to-image (LayoutDiffusion) modules, and enforce local and global shadow-semantic alignment via two diffusion-GAN discriminators.
Key results
- SGT+ reduces object rAVG on VG from 10.51 to 8.93 and raises mIoU on VG from 22.7 to 28.4.
- FID on VG improves from 20.12 (SGT) to 19.47 (SGT+) and on COCO-stuff from 19.12 to 18.38.
- In user studies with 91 participants, 89% of predicted layouts were judged plausible and 87% of SGT+ outputs were preferred for visual authenticity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the shadow-enlightened module to SGT | SGT · Visual Genome (VG) | mIoU: 22.7 -> 28.4 | Helped |
| Add the shadow-enlightened module to SGT | SGT · COCO-stuff | mIoU: 32.6 -> 38.1 | Helped |
| Add the shadow-enlightened module to SGT | SGT · Visual Genome (VG) | FID: 20.12 -> 19.47 | Helped |
| Add the shadow-enlightened module to SGT | SGT · COCO-stuff | FID: 19.12 -> 18.38 | Helped |
| Add semantic removal to the shadow representation (ablation) | SGT shadow-enlightened pipeline · Visual Genome (VG) | mIoU: 22.7 -> 25.4 | Helped |
| Add illumination feature concatenation (ablation) | SGT shadow-enlightened pipeline · Visual Genome (VG) | mIoU: 25.4 -> 26.4 | Helped |
| Add the local alignment discriminator (ablation) | SGT shadow-enlightened pipeline · Visual Genome (VG) | FID: 19.92 -> 19.48 | Helped |
| Add the global consistency discriminator (ablation) | SGT shadow-enlightened pipeline · COCO-stuff | FID: 18.93 -> 18.59 | Helped |
| Add the local alignment discriminator (ablation) | SGT shadow-enlightened pipeline · COCO-stuff | FID: 19.02 -> 18.60 | Helped |
| Add the global consistency discriminator (ablation) | SGT shadow-enlightened pipeline · Visual Genome (VG) | FID: 19.89 -> 19.52 | Helped |
Limitations
- Relies on pretrained shadow detection, instance shadow detection, and shadow removal models; the instance shadow detector cannot detect shadows of unobserved objects.
- Layout expansion and layout-to-image modules are not the focus and are replaceable; evaluation only uses SGT as the base pipeline.
- Training is stopped manually and no formal stopping criterion is reported; evaluation is limited to VG and COCO-stuff at 256x256.
Experimental setup
Datasets: Visual Genome COCO-stuff
Benchmarks: Visual Genome COCO-stuff
Metrics: rAVG Hit@1/3 mIoU FID
Baselines: SGT
Builds on: SGT GTwE LayoutDiffusion Diffusion-GAN RePaint
Closest papers by meaning
- OmniSR: Shadow Removal Under Direct and Indirect Lighting (AAAI 2025 · similarity 0.86)
- MetaShadow: Object-Centered Shadow Detection, Removal, and Synthesis (CVPR 2025 · similarity 0.86)
- DeS3: Adaptive Attention-Driven Self and Soft Shadow Removal Using ViT Similarity (AAAI 2024 · similarity 0.85)
- When Shadow Removal Meets Intrinsic Image Decomposition: A Joint Learning Framework Using Unpaired Data (AAAI 2025 · similarity 0.85)
- Detail-Preserving Latent Diffusion for Stable Shadow Removal (CVPR 2025 · similarity 0.85)
- Recasting Regional Lighting for Shadow Removal (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.