Research map / Image generation / Image restoration
Detail-Preserving Latent Diffusion for Stable Shadow Removal
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts a pre-trained Stable Diffusion model for mask-free shadow removal via two stages: latent-space denoiser fine-tuning with a fixed VAE, then a shadow-aware detail injection module that modulates the VAE decoder with encoder features and DINO v2 features.
Problem
Shadow removal methods overfit limited training data and generalize poorly; applying Stable Diffusion to shadow removal loses high-frequency detail due to lossy VAE latent compression.
Approach
Fine-tune only the SD denoiser in latent space with a frozen VAE (z0-prediction), then add a Detail Injection module that feeds VAE encoder features and DINO v2 features into RRDB blocks to modulate the frozen VAE decoder, restoring shadow-free details without a shadow mask.
Key results
- Achieves 35.19 PSNR on ISTD+ and 30.56 PSNR on INS, the highest among compared methods on INS and best mask-free result on ISTD+.
- Cross-dataset evaluation: 26.23 PSNR on ISTD+→SRD vs. 25.86 for OmniSR, and 30.23 on SRD→ISTD+ vs. 29.48 for ShadowDiffusion.
- Stage two raises ISTD+ PSNR from 29.95 (stage one) to 35.02-35.19; z0-prediction reduces seed variance from 0.239 to 0.146 in stage one.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use z0-prediction instead of ϵ-prediction in the latent diffusion stage (ablation) | Ours · ISTD+ | PSNR: 29.66 -> 29.95 | Helped |
| Use z0-prediction instead of ϵ-prediction in the latent diffusion stage (ablation) | Ours · ISTD+ | PSNR variance: 0.239 -> 0.146 | Helped |
| Add the detail injection stage after latent diffusion (ablation) | Ours · ISTD+ | PSNR: 29.95 -> 35.19 | Helped |
| Add the detail injection stage after latent diffusion (ablation) | Ours · SRD | PSNR: 28.18 -> 33.63 | Helped |
| Add the detail injection stage after latent diffusion (ablation) | Ours · INS | PSNR: 28.78 -> 30.56 | Helped |
| Use Stable Diffusion pre-training (ablation) | Ours · ISTD+ | PSNR: 33.38 -> 35.19 | Helped |
| Use Stable Diffusion pre-training (ablation) | Ours · SRD | PSNR: 32.46 -> 33.63 | Helped |
| Use Stable Diffusion pre-training (ablation) | Ours · INS | PSNR: 29.55 -> 30.56 | Helped |
| Add DINO features to the detail injection model (ablation) | Ours · ISTD+ | PSNR: 34.76 -> 35.19 | Helped |
| Add DINO features to the detail injection model (ablation) | Ours · SRD | PSNR: 33.04 -> 33.63 | Helped |
| Add DINO features to the detail injection model (ablation) | Ours · INS | PSNR: 30.30 -> 30.56 | Helped |
| Use the full two-stage method instead of stage one only (ablation) | Ours · ISTD+ | PSNR variance: 0.146 -> 0.160 | Helped |
Surprising findings
- The pre-trained VAE handles shadow-free images without being fine-tuned, so the VAE is kept frozen despite not being trained for this purpose.
- Mask-free method outperforms several mask-based methods on ISTD+ and INS, without requiring shadow masks.
Limitations
- May miss shadows in complex scenes such as self-shadows on objects.
- Focuses on color variation correction and cannot fully preserve exposure in non-shadow areas.
- Lower PSNR than DeS3 on highly textured regions such as sand and grass on SRD.
Experimental setup
Datasets: SRD ISTD+ WSRD+ INS
Benchmarks: ISTD+ SRD INS WSRD+
Metrics: PSNR SSIM PSNR variance across seeds
Baselines: AutoExposure Zhu et al. BMNet ShadowFormer DMTN ShadowDiffusion HomoFormer InstanceShadow TBRNet Refusion DeS3 OmniSR
Builds on: Stable Diffusion Lotus DDIM RRDB
Closest papers by meaning
- Diff-Shadow: Global-guided Diffusion Model for Shadow Removal (AAAI 2025 · similarity 0.90)
- Polarization Guided Mask-Free Shadow Removal (AAAI 2025 · similarity 0.87)
- When Shadow Removal Meets Intrinsic Image Decomposition: A Joint Learning Framework Using Unpaired Data (AAAI 2025 · similarity 0.87)
- In-N-Out: Lifting 2D Diffusion Prior for 3D Object Removal via Tuning-Free Latents Alignment (NeurIPS 2024 · similarity 0.87)
- DeS3: Adaptive Attention-Driven Self and Soft Shadow Removal Using ViT Similarity (AAAI 2024 · similarity 0.86)
- OmniSR: Shadow Removal Under Direct and Indirect Lighting (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.