Research map / Image generation / Autoregressive and latent generation
Shadow Generation for Composite Image Using Diffusion Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts ControlNet on Stable Diffusion with an intensity modulation module and post-processing for foreground shadow generation, and contributes the larger DESOBAv2 dataset built via object-shadow detection and inpainting.
Problem
Generating realistic foreground shadows for inserted objects in composite images; prior image-to-image GAN methods trained on scarce paired data produce shadows with inaccurate shapes and intensities.
Approach
Repurposes a pretrained ControlNet/Stable Diffusion foundation model for shadow generation, adding a weighted noise loss on expanded shadow regions, an intensity encoder that predicts channel-wise scale/bias to modulate noise within a predicted foreground shadow mask, and a post-processing network to fix color shift and background variation; the training data is expanded to DESOBAv2 via an automat
Key results
- On DESOBAv2 BOS test images, SGDiffusion achieves GRMSE 6.098 and LRMSE 53.611, versus 7.184 and 68.255 for SGRNet.
- Ablation shows intensity modulation and weighted noise loss together reduce global RMSE from 8.285 to 6.098 after post-processing.
- A user study with 50 raters on real composite images compares SGDiffusion against baselines via Bradley-Terry scores.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SGDiffusion instead of SGRNet | DESOBAv2 BOS test images · DESOBAv2 BOS test images | global RMSE: 7.184 -> 6.098 | Helped |
| Use SGDiffusion instead of SGRNet | DESOBAv2 BOS-free test images · DESOBAv2 BOS-free test images | global RMSE: 15.596 -> 15.110 | Helped |
| Add weighted noise loss (ablation) | ControlNet on DESOBAv2 BOS test images · DESOBAv2 BOS test images | global RMSE: 8.285 -> 7.041 | Helped |
| Do not expand the foreground shadow mask for weighted loss (ablation) | SGDiffusion on DESOBAv2 BOS test images · DESOBAv2 BOS test images | global RMSE: 7.041 -> 8.319 | Hurt |
| Add intensity modulation (ablation) | ControlNet on DESOBAv2 BOS test images · DESOBAv2 BOS test images | global RMSE: 8.285 -> 7.357 | Helped |
| Include the background shadow mask as intensity-encoder input (ablation) | SGDiffusion on DESOBAv2 BOS test images · DESOBAv2 BOS test images | global RMSE: 7.410 -> 7.357 | Helped |
| Add post-processing (ablation) | SGDiffusion on DESOBAv2 BOS test images · DESOBAv2 BOS test images | global RMSE: 13.447 -> 6.098 | Helped |
Surprising findings
- Local SSIM for SGDiffusion on BOS test images (0.370) is higher than baselines, but its LS on BOS-free images (0.117) is below ARShadowGAN's? No: it is slightly above SGRNet, yet the post-processing removal raises global RMSE from 6.098 to 13.447, showing that post-processing is essential.
Limitations
- Relies on a pretrained object-shadow detector and inpainting model, so dataset quality depends on them and includes manual filtering.
- Inpainted deshadowed backgrounds can retain artifacts or undesirable shadows, requiring manual filtering.
Experimental setup
Datasets: DESOBAv2 DESOBA
Benchmarks: DESOBAv2 DESOBA
Metrics: GR LR GS LS GB LB user preference (Bradley-Terry)
Baselines: ShadowGAN MaskShadowGAN ARShadowGAN SGRNet
Builds on: ControlNet Stable Diffusion DESOBA
Closest papers by meaning
- Shadow Generation Using Diffusion Model with Geometry Prior (CVPR 2025 · similarity 0.90)
- Shadow Generation with Decomposed Mask Prediction and Attentive Shadow Filling (AAAI 2024 · similarity 0.89)
- Detail-Preserving Latent Diffusion for Stable Shadow Removal (CVPR 2025 · similarity 0.85)
- Diff-Shadow: Global-guided Diffusion Model for Shadow Removal (AAAI 2025 · similarity 0.85)
- MetaShadow: Object-Centered Shadow Detection, Removal, and Synthesis (CVPR 2025 · similarity 0.84)
- Gaussian Shadow Casting for Neural Characters (CVPR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.