Research map / Image generation / Editing and style transfer
PS-Diffusion: Photorealistic Subject-Driven Image Editing with Disentangled Control and Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PS-Diffusion, a ControlNet-and-Stable-Diffusion-based subject-driven editor that disentangles foreground lighting (via intrinsic decomposition) and background effects (via an effect map branch) and adds an effect attention module, trained on a new Blender-rendered Replace-5K dataset.
Problem
Subject-driven image editing that inserts a reference object into a scene often yields non-photorealistic results because contextual interactions such as reflections, illumination, and shadows are not simulated consistently with the merged object.
Approach
Split contextual effects by affected region: foreground lighting is conditioned via intrinsic image decomposition features, background effects (shadows, reflections) are predicted as an effect map by an additional ControlNet-based branch, and an Effect Attention Module separates interaction learning from subject learning in the denoising U-Net.
Key results
- On Replace-5K, PS-Diffusion reaches PSNR 22.583, LPIPS 0.174 and FID 5.109, beating the best baseline Remove-Insert (PSNR 21.490, FID 6.241).
- On DreamBooth and DreamEdit, PS-Diffusion obtains CLIP-I 0.845 and 0.851 and FID 22.919 and 23.383, best FID among compared methods.
- Ablation on Replace-5K shows FECB, BECB and EAM each improve PSNR, from 21.480 baseline to 22.583 with all modules.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PS-Diffusion instead of Remove-Insert | subject-driven image editing · Replace-5K test | PSNR: 21.490 -> 22.583 | Helped |
| Use PS-Diffusion instead of Remove-Insert | subject-driven image editing · Replace-5K test | FID: 6.241 -> 5.109 | Helped |
| Use PS-Diffusion instead of Remove-Insert | subject-driven image editing · DreamBooth | FID: 23.607 -> 22.919 | Helped |
| Use PS-Diffusion instead of Remove-Insert | subject-driven image editing · DreamEdit | FID: 24.191 -> 23.383 | Helped |
| Add FECB (ablation) | PS-Diffusion without effect modules · Replace-5K test | PSNR: 21.480 -> 21.799 | Helped |
| Add BECB to FECB (ablation) | PS-Diffusion with FECB · Replace-5K test | PSNR: 21.799 -> 22.010 | Helped |
| Add EAM to FECB + BECB (ablation) | PS-Diffusion with FECB and BECB · Replace-5K test | PSNR: 22.010 -> 22.583 | Helped |
| Replace EAM with direct feature weighting (ablation) | PS-Diffusion with FECB and BECB · Replace-5K test | PSNR: 22.583 -> 22.010 | Hurt |
| Increase the EAM coefficient excessively (ablation) | PS-Diffusion · Replace-5K test | PSNR | Hurt |
| Add the LAB-derived rough effect mask (ablation) | BECB effect-map prediction · Replace-5K test | effect-map accuracy | Helped |
Surprising findings
- Remove-Insert two-stage pipeline achieves CLIP-I and DINO-I close to PS-Diffusion on real data, because harmonization keeps the foreground largely unchanged, though it has higher FID due to copy-paste artifacts.
Limitations
- Fails on hard samples with complex lighting or strong specular reflections (FID 8.781 on hard samples vs 5.109 overall).
- Failure cases in mirrors where edited objects are ambiguous or unedited.
- Relies on a lighting estimator and intrinsic decomposition that lack priors for reflective or diffuse materials.
Experimental setup
Datasets: Replace-5K DreamBooth DreamEdit Open Images V7
Benchmarks: Replace-5K DreamBooth DreamEdit
Metrics: PSNR CLIP DINO LPIPS FID CLIP-I DINO-I
Baselines: IP-Adapter Paint-by-Example ObjectStitch AnyDoor Remove-Insert (PowerPaint + harmonization)
Builds on: ControlNet Stable Diffusion AnyDoor Paint-by-Example DINOv2
Closest papers by meaning
- PAIR Diffusion: A Comprehensive Multimodal Object-Level Image Editor (CVPR 2024 · similarity 0.89)
- Diffusion Handles Enabling 3D Edits for Diffusion Models by Lifting Activations to 3D (CVPR 2024 · similarity 0.88)
- Adapting Diffusion Models for Improved Prompt Compliance and Controllable Image Synthesis (NeurIPS 2024 · similarity 0.88)
- Towards Efficient Diffusion-Based Image Editing with Instant Attention Masks (AAAI 2024 · similarity 0.88)
- An Item Is Worth a Prompt: Versatile Image Editing with Disentangled Control (AAAI 2025 · similarity 0.88)
- Move Anything with Layered Scene Diffusion (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.