Research map / Video / Video generation
Re-HOLD: Video Hand Object Interaction Reenactment via adaptive Layout-instructed Diffusion Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Re-HOLD reenacts human-object interaction videos by conditioning a Stable Diffusion-based video diffusion model on layout boxes for hands and objects, hand-mesh control, memory banks, and an adaptive layout adjustment for cross-object reenactment.
Problem
Generating realistic human-object interaction (HOI) videos where hands and objects are in contact, especially when the target object differs in size and shape from the original object in the driving video.
Approach
Uses specialized layout representations (fixed-size square hand boxes plus a variable object box) to disentangle hand and object modeling, adds hand and object memory-bank attention for texture restoration, and adaptively adjusts layout boxes at inference using a contact-distance rule to avoid floating or implausible contacts for differently sized objects.
Key results
- Self-reenactment on HOI4D: Re-HOLD FID 26.583 vs HOI-Swap 30.152 and VideoSwap 158.964; PSNR 31.984 vs 31.528 for HOI-Swap.
- In-house self-reenactment: Re-HOLD PSNR 33.451 and FID 19.021, versus RealisDance FID 26.337 and AnimateAnyone FID 26.361.
- User study: Re-HOLD scored 0.92 on HOI consistency and 0.92 on object consistency, above HOI-Swap at 0.76 and 0.40.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add adaptive layout adjustment strategy for cross-object reenactment (ablation) | Re-HOLD · HOI4D cross-reenactment | subject consistency: 0.952 -> 0.955 | Helped |
| add hand attention (hand memory bank) layers (ablation) | Re-HOLD · HOI4D self-reenactment | hand agreement: 0.756 -> 0.773 | Helped |
| add object attention (object memory bank) layers (ablation) | Re-HOLD · HOI4D self-reenactment | hand agreement: 0.767 -> 0.773 | Helped |
| add layout guidance (hand and object boxes) instead of raw 3D hand mesh renderings (ablation) | Re-HOLD · HOI4D self-reenactment | hand agreement: 0.753 -> 0.773 | Helped |
| add layout guidance instead of raw 3D hand mesh renderings (ablation) | Re-HOLD · HOI4D cross-reenactment | subject consistency: 0.950 -> 0.955 | Helped |
| full Re-HOLD vs. baselines (layout, HOI restoration, adaptive strategy) | Re-HOLD · HOI4D cross-reenactment | FID: 26.583 vs 30.152 (HOI-Swap) | Helped |
| full Re-HOLD vs. baselines | Re-HOLD · HOI4D self-reenactment | FID: 19.021 vs 26.337 (RealisDance) and 26.361 (AnimateAnyone) | Mixed |
Limitations
- Dataset focuses on hand movements for object display in live-streaming, so 3D object manipulation such as multi-view generation performs poorly.
- Hand agreement, PSNR and FID are only computable for self-reenactment because cross-reenactment lacks ground truth.
- Object detection errors make the contact-agreement metric unreliable, so it was excluded.
Experimental setup
Datasets: HOI4D in-house collected subject-object videos
Benchmarks: in-house HOI dataset HOI4D
Metrics: hand agreement hand fidelity subject consistency motion smoothness PSNR FID user preference
Baselines: AnyV2V VideoSwap AnimateAnyone RealisDance HOI-Swap
Builds on: Stable Diffusion AnimateAnyone ControlNet HOI-Swap HaMeR
Closest papers by meaning
- SViMo: Synchronized Diffusion for Video and Motion Generation in Hand-object Interaction Scenarios (NeurIPS 2025 · similarity 0.90)
- InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generation (CVPR 2025 · similarity 0.90)
- HOI-Swap: Swapping Objects in Videos with Hand-Object Interaction Awareness (NeurIPS 2024 · similarity 0.89)
- Text2HOI: Text-guided 3D Motion Generation for Hand-Object Interaction (CVPR 2024 · similarity 0.89)
- Hand-Centric Motion Refinement for 3D Hand-Object Interaction via Hierarchical Spatial-Temporal Modeling (AAAI 2024 · similarity 0.88)
- HandDiffuse: Generative Controllers for Two-Hand Interactions via Diffusion Models (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.