Research map / Video / Video generation
HOI-Swap: Swapping Objects in Videos with Hand-Object Interaction Awareness
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HOI-Swap, a two-stage self-supervised diffusion framework that swaps hand-held objects in videos using one reference image, with an image inpainting stage for HOI-aware single-frame edits and a video stage conditioned on optical-flow-warped points for controllable motion alignment.
Problem
Swapping in-contact objects in videos while keeping realistic hand-object interactions and following the source motion, which existing image and video editing diffusion models handle poorly when object shape or function changes.
Approach
A two-stage pipeline: stage I learns HOI-aware object swapping in one frame by inpainting a square-bbox-masked frame conditioned on an augmented reference object (DINO features); stage II propagates the edit through the video, conditioned on a video warped from the edited anchor frame using a tunable fraction of RAFT-tracked points, giving controllable motion alignment.
Key results
- Image editing: HOI-Swap reaches 87.9 contact agreement, 79.8 hand agreement, 97.4 hand fidelity, and 72.1% user preference vs. AnyDoor's 15.6%.
- Video editing: HOI-Swap achieves 93.1 contact agreement, 78.6 hand agreement, 97.6 hand fidelity and 86.4% user preference, versus VideoSwap's 1.2% preference.
- Stage-II with SAM-2 estimated boxes drops contact agreement from 89.9 to 87.8 but still exceeds prior best on HOI metrics.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HOI-Swap instead of the strongest image-editing baseline | Image editing · Image editing | user preference: 15.6 -> 72.1 | Helped |
| Use HOI-Swap instead of the strongest video-editing baseline | Video editing · Video editing | user preference: 1.2 -> 86.4 | Helped |
| Replace the CLIP encoder with DINO in stage II (ablation) | HOI-Swap · Video editing | contact agreement: 89.9 -> 88.9 | No clear effect |
| Replace ground-truth boxes with SAM-2 estimated boxes (ablation) | HOI-Swap · Video editing | hand agreement: 79.0 -> 73.9 | Hurt |
| Use the original object segmentation mask instead of a square bounding-box mask (ablation) | Stage-I HOI-Swap · Image editing | HOI grasp realism | Hurt |
| Select a hand-object contact frame for stage-I editing instead of a non-contact frame (ablation) | HOI-Swap · Video editing | subsequent-frame interaction realism | Helped |
| Use one-stage generation instead of the two-stage pipeline (ablation) | HOI-Swap · Video editing | reference-object identity | Hurt |
| Sample all motion points instead of no points (ablation) | HOI-Swap · Video editing | motion alignment | Helped |
| Use 50% of motion points rather than no points or all points (ablation) | HOI-Swap · Video editing | plausible motion | Helped |
Surprising findings
- Baseline methods, including text-guided approaches, performed notably poorly on in-contact object swapping despite official code and tuning.
- Using a DINO encoder in stage II performs similarly to CLIP (subject consistency 91.2 vs 91.4), suggesting encoder choice matters little there.
Limitations
- Struggles to generalize to very different unseen reference objects (e.g., scissors not in training data).
- Designed for short clips where hand-object interaction is stable; longer sequences with multiple interactions are not handled.
- Controllability is limited to point sparsity with no spatial region control.
Experimental setup
Datasets: HOI4D Ego-Exo4D EPIC-Kitchens TCN Pouring
Benchmarks: HOI4D held-out Ego-Exo4D validation EPIC-Kitchens (zero-shot) TCN Pouring (zero-shot)
Metrics: contact agreement hand agreement hand fidelity subject consistency motion smoothness user preference rate
Baselines: Paint by Example AnyDoor Affordance Diffusion Per-frame AnyDoor AnyV2V VideoSwap
Builds on: Stable Diffusion Stable Video Diffusion AnyDoor AnyV2V RAFT
Closest papers by meaning
- Re-HOLD: Video Hand Object Interaction Reenactment via adaptive Layout-instructed Diffusion Model (CVPR 2025 · similarity 0.89)
- SViMo: Synchronized Diffusion for Video and Motion Generation in Hand-object Interaction Scenarios (NeurIPS 2025 · similarity 0.89)
- HO-Cap: A Capture System and Dataset for 3D Reconstruction and Pose Tracking of Hand-Object Interaction (NeurIPS 2025 · similarity 0.88)
- InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generation (CVPR 2025 · similarity 0.87)
- VideoSwap: Customized Video Subject Swapping with Interactive Semantic Point Correspondence (CVPR 2024 · similarity 0.87)
- FlowHOI: Flow-based Semantics-Grounded Generation of Hand-Object Interactions for Dexterous Robot Manipulation (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.