Research map / Video / Video generation
VideoHandles: Editing 3D Object Compositions in Videos Using Video Generative Priors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Training-free method that edits the 3D position of an object in a video of a static scene by warping OpenSora video-model features through a DUSt3R 3D reconstruction and using them as energy guidance during generation.
Problem
Editing 3D object composition (position) in videos of static scenes with camera motion while keeping temporal consistency, identity, and plausible effects such as shadows and reflections; prior video editing work only handles appearance or motion.
Approach
Lift intermediate spatial self-attention features of a flow-based video generator to a shared 3D reconstruction, transform the selected object in 3D, project the warped features back to each frame, and guide the generative process with an energy term weighted by self-attention maps and with null-text prediction in the original object region.
Key results
- Frame LPIPS of 3.71 for VideoHandles versus 18.63 for DiffusionHandles (x10^2), indicating much better temporal consistency.
- User study: 100% preference for plausibility over DiffusionHandles, and 75% and 57% preference for identity preservation and edit coherence versus SDEdit-refined direct warping.
- Ablations show full method outperforms w/ temporal features, w/o self-attention weighting, and w/o null-text in user study and Frame LPIPS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VideoHandles instead of DiffusionHandles | Video composition editing · 27 generated videos | Frame LPIPS (×10²): 18.63 -> 3.71 | Helped |
| Use VideoHandles instead of direct warping | Video composition editing · 27 generated videos | Frame LPIPS (×10²): 5.19 -> 3.71 | Helped |
| Use VideoHandles instead of direct warping refined with SDEdit | Video composition editing · 27 generated videos | Frame LPIPS (×10²): 5.03 -> 3.71 | Helped |
| Add temporal self-attention features (ablation) | VideoHandles · 27 generated videos | Frame LPIPS (×10²): 3.71 -> 3.81 | Hurt |
| Add self-attention-based weighting (ablation) | VideoHandles · 27 generated videos | Frame LPIPS (×10²): 3.77 -> 3.71 | Helped |
| Add null-text prediction in the original object area (ablation) | VideoHandles · 27 generated videos | Frame LPIPS (×10²): 3.79 -> 3.71 | Helped |
Surprising findings
- Direct frame warping produces noisy videos due to camera and reconstruction errors, so warping features instead of pixels is necessary.
- Temporal self-attention features produce global changes unsuitable for local object edits, so only spatial-layer features are used.
Limitations
- Performance constrained by current video generative model capabilities.
- Limited to videos of stationary (static) scenes; dynamic scenes not handled.
- Relies on accuracy of 3D reconstruction and camera pose estimation, causing warping errors.
Experimental setup
Datasets: RealEstate10K
Benchmarks: in-house set of 27 generated 51-frame 320x320 videos
Metrics: Frame LPIPS user preference for plausibility, identity preservation, edi
Baselines: DiffusionHandles Direct frame warping Direct warping + SDEdit
Builds on: DiffusionHandles OpenSora DUSt3R energy-guided generative process null-text inversion SDEdit
Closest papers by meaning
- ObjectMover: Generative Object Movement with Video Prior (CVPR 2025 · similarity 0.86)
- Pathways on the Image Manifold: Image Editing via Video Generation (CVPR 2025 · similarity 0.85)
- DreamScene4D: Dynamic Multi-Object Scene Generation from Monocular Videos (NeurIPS 2024 · similarity 0.85)
- Instant3dit: Multiview Inpainting for Fast Editing of 3D Objects (CVPR 2025 · similarity 0.85)
- Generative Omnimatte: Learning to Decompose Video into Layers (CVPR 2025 · similarity 0.84)
- Interactive3D: Create What You Want by Interactive 3D Generation (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.