Research map / Video / Video generation

The Surprising Effectiveness of Video Diffusion Models for Hand Motion Reconstruction

NeurIPS 2026 · Video › Video generation · approach: Trajectory-guided motion control · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Adapts a pretrained video diffusion model (Wan2.1-VACE) via hand-overlay rendering and decodes 4D two-hand MANO pose with a dual-branch decoder, without detectors or test-time optimization.

Problem

4D two-hand motion reconstruction from egocentric video under heavy occlusion, where detector-based image methods fail and video methods rely on hand-only temporal priors learned from scarce hand-pose labels.

Approach

Use internal features of a pretrained video diffusion transformer, specialized to hands by fine-tuning only its VACE branch to regenerate frames with rendered hand overlays, and decode MANO pose plus image coordinates with a dual-branch decoder whose camera translation is split into regressed depth and closed-form in-plane shift.

Key results

• On ARCTIC, frame accuracy reaches 0.997 vs 0.919 for the best baseline WiLoR, and jitter drops to 3.18 vs 12.8 for Dyn-HaMR.
• On HOT3D, MPJPE-p improves from about 31.0 mm (WiLoR) to 21.5 mm.
• On held-out HOI4D, ViDiHand ranks first on eight of nine metrics, with EPE-p 24.5 px vs 41.6 px and jitter 4.0 vs 17.4.

Surprising findings

• Pretrained text-to-video backbone without hand adaptation already gets strong detection (FAcc 0.9932), and a randomly initialized transformer lags all pretrained backbones.
• DINOv3-H+ (840M image backbone) underperforms the 1.3B Wan backbone on all metrics, with the largest gap in jitter.

Limitations

• Runs at 5.5 fps on 4 A100 GPUs, so it is currently an offline annotation tool rather than real-time.
• Training still relies on pose-labeled video; the video backbone's pretraining corpus is undisclosed, so held-out status on HOI4D cannot be audited.
• Does not beat all baselines on every metric; e.g., CT-p on HOI4D and MPJPE-p/ hand pose trade-offs with the solve ablation.

Experimental setup

Datasets: ARCTIC HOT3D EgoDex HOI4D

Benchmarks: ARCTIC HOT3D HOI4D

Metrics: FAcc Recall F1 MPJPE-p PA-MPJPE-p EPE-p GO-p CT-p Jitter

Baselines: HaMeR WiLoR Hamba InterWild WildHands OmniHands Dyn-HaMR HaWoR

Builds on: Wan2.1-VACE MANO HaMeR HaWoR DIFT

Closest papers by meaning

• FoundHand: Large-Scale Domain-Specific Learning for Controllable Hand Image Generation (CVPR 2025 · similarity 0.89)
• SViMo: Synchronized Diffusion for Video and Motion Generation in Hand-object Interaction Scenarios (NeurIPS 2025 · similarity 0.89)
• Dyn-HaMR: Recovering 4D Interacting Hand Motion from a Dynamic Camera (CVPR 2025 · similarity 0.88)
• HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Video (CVPR 2024 · similarity 0.88)
• Re-HOLD: Video Hand Object Interaction Reenactment via adaptive Layout-instructed Diffusion Model (CVPR 2025 · similarity 0.88)
• ComPose: When to Trust Hands for Object Pose Tracking (NeurIPS 2026 · similarity 0.88)