Research map / Image generation / Editing and style transfer
Diffusion Handles Enabling 3D Edits for Diffusion Models by Lifting Activations to 3D
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Training-free method that lifts Stable Diffusion decoder activations to a 3D point cloud using estimated depth, applies a rigid 3D edit, projects them back, and uses them as guidance to regenerate an edited image with 3D-aware object edits.
Problem
Enabling 3D-aware object-level edits (translation, rotation, scale) on images from diffusion models or real photos while preserving object identity and producing plausible lighting, shadows and occlusions, without fine-tuning or 3D data.
Approach
Lift intermediate diffusion decoder activations of a selected object to 3D using a proxy depth map, apply a 3D rigid transform to the depth and activations, project them back to image space as a warp flow, and use the warped activations as energy guidance during depth-conditioned diffusion sampling.
Key results
- On the 50-edit synthetic Benchmark, Diffusion Handles reach edit-adherence IoU 0.85 versus 0.52 for Zero123, 0.37 for ObjectStitch and 0.15 for 3DIT.
- Identity preservation on the Benchmark: L1 cycle error 0.71 (x10) and LPIPS 0.19, better than Zero123 (1.05, 0.31), 3DIT (0.74, 0.27) and ObjectStitch (0.89, 0.25).
- In a 22-user study, users preferred the method over baselines on plausibility, identity preservation and edit adherence.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use local foreground guidance instead of average foreground guidance (ablation) | Diffusion Handles · Ablation Study | identity preservation | Helped |
| Use local foreground guidance instead of average foreground guidance (ablation) | Diffusion Handles · Ablation Study | edit adherence | Helped |
| Stop guidance before the final denoising steps (ablation) | Diffusion Handles · Ablation Study | plausibility | Helped |
| Guide too few denoising steps (ablation) | Diffusion Handles · Ablation Study | identity preservation | Hurt |
| Guide only decoder layer 2 instead of using the cyclic layer schedule (ablation) | Diffusion Handles · Ablation Study | identity preservation | Hurt |
| Guide only decoder layer 2 instead of using the cyclic layer schedule (ablation) | Diffusion Handles · Ablation Study | edit adherence | Hurt |
| Guide only decoder layer 3 instead of using the cyclic layer schedule (ablation) | Diffusion Handles · Ablation Study | identity preservation | Hurt |
| Guide only decoder layer 3 instead of using the cyclic layer schedule (ablation) | Diffusion Handles · Ablation Study | edit adherence | Helped |
| Guide layers 2 and 3 simultaneously rather than cyclically (ablation) | Diffusion Handles · Ablation Study | artifacts | Hurt |
| Use a cyclic schedule for decoder layers 2 and 3 (ablation) | Diffusion Handles · Ablation Study | artifacts | Helped |
| Set background guidance weight low relative to foreground guidance weight (ablation) | Diffusion Handles · Ablation Study | foreground lighting preservation | Helped |
| Set background guidance weight low relative to foreground guidance weight (ablation) | Diffusion Handles · Ablation Study | background lighting preservation | Hurt |
Surprising findings
- Some users found edited results more plausible than the original input images in certain cases.
Limitations
- Large edits that expose strong distortions of the estimated depth or regions hidden in the original view give low-quality outputs.
- Identity preservation is imperfect compared with ideal.
- Only handles rigid translations, rotations and scaling; no video extension yet.
Experimental setup
Datasets: ModelNet40 PhotoGen (photos and generated images)
Benchmarks: PhotoGen Benchmark (synthetic, 50 edits)
Metrics: IoU (edit adherence) L1 cycle-consistency (identity) LPIPS cycle-consistency (identity) user preference
Baselines: 3DIT Zero123 ObjectStitch Diffusion Self-Guidance (ablation)
Builds on: Stable Diffusion v2 Null-Text Inversion Diffusion Self-Guidance Zero123 SAM
Closest papers by meaning
- MirrorVerse: Pushing Diffusion Models to Realistically Reflect the World (CVPR 2025 · similarity 0.89)
- PAIR Diffusion: A Comprehensive Multimodal Object-Level Image Editor (CVPR 2024 · similarity 0.88)
- PS-Diffusion: Photorealistic Subject-Driven Image Editing with Disentangled Control and Attention (CVPR 2025 · similarity 0.88)
- Generating Images with 3D Annotations Using Diffusion Models (ICLR 2024 · similarity 0.88)
- Diffusion Renderer: Neural Inverse and Forward Rendering with Video Diffusion Models (CVPR 2025 · similarity 0.88)
- SIGNeRF: Scene Integrated Generation for Neural Radiance Fields (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.