Research map / Video / Video generation
ManiVideo: Generating Hand-Object Manipulation Video with Dexterous and Generalizable Grasping
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ManiVideo, a diffusion-based framework that generates bimanual hand-object manipulation videos from MANO hand and object motion, using multi-layer occlusion maps and Objaverse-based object training.
Problem
Generating temporally consistent hand-object manipulation videos with plausible finger self-occlusion and mutual hand-object occlusion, and generalizing to diverse unseen objects despite scarce HOI video data.
Approach
A multi-layer occlusion (MLO) representation of occlusion-free per-part normal maps plus occlusion confidence (depth) maps, embedded into an Animate Anyone-style UNet both at the initial noise and via cross-attention in added transformer blocks; plus a two-stage training strategy that mixes Objaverse object-only data with HOI and human video data.
Key results
- On DexYCB, ManiVideo reaches FID 49.96 and LPIPS 0.079, versus 53.95 FID for ADiff and 64.74 for HOGAN.
- On the self-collected dataset, ManiVideo reports FID 37.70 and MPJPE 32.89, improving over ADiff (39.91 FID, 37.45 MPJPE).
- Ablation shows removing Objaverse raises FID from 37.70 to 61.60 and MPJPE from 32.89 to 37.33.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the MLO representation instead of basic 2D depth conditions (ablation) | ManiVideo · Our Dataset | FID: 46.67 -> 37.70 | Helped |
| embed MLO into both initial noise and added transformer blocks rather than only initial noise (ablation) | ManiVideo · Our Dataset | FID: 40.60 -> 37.70 | Helped |
| add Objaverse training data (ablation) | ManiVideo · Our Dataset | FID: 61.60 -> 37.70 | Helped |
| use depth confidence maps rather than no confidence map (ablation) | ManiVideo · HOI generation | finger alignment | Helped |
| replace depth confidence maps with learnable maps predicted from normal maps (ablation) | ManiVideo · HOI generation | 3D consistency | Hurt |
| replace depth confidence maps with soft aggregation of confidence and normal maps (ablation) | ManiVideo · HOI generation | 3D consistency | Hurt |
| use ManiVideo rather than ADiff | hand-object video generation · DexYCB | FID: 53.95 -> 49.96 | Helped |
| use ManiVideo rather than ADiff | hand-object video generation · Our Dataset | MPJPE: 37.45 -> 32.89 | Helped |
Limitations
- Experiments are conducted in a laboratory environment, and driving motions are captured rather than generated.
- Fails on unreasonable driving signals for open-set objects.
- Evaluation of HOGAN limited to single-hand DexYCB setting; the self-collected dataset is not released.
Experimental setup
Datasets: Objaverse DexYCB Human4DiT TikTok
Benchmarks: DexYCB self-collected bimanual HOI video dataset
Metrics: FID LPIPS PSNR SSIM MPJPE
Baselines: HOGAN Affordance Diffusion (ADiff) ControlNet with Stable Diffusion v1.5 (CDiff)
Builds on: Animate Anyone Stable Diffusion ControlNet Objaverse
Closest papers by meaning
- ComPose: When to Trust Hands for Object Pose Tracking (NeurIPS 2026 · similarity 0.89)
- LatentHOI: On the Generalizable Hand Object Motion Generation with Latent Hand Diffusion. (CVPR 2025 · similarity 0.88)
- FlowHOI: Flow-based Semantics-Grounded Generation of Hand-Object Interactions for Dexterous Robot Manipulation (NeurIPS 2026 · similarity 0.88)
- SViMo: Synchronized Diffusion for Video and Motion Generation in Hand-object Interaction Scenarios (NeurIPS 2025 · similarity 0.88)
- HandEdit: A Unified Benchmark for Egocentric Human-to-Robot Dexterous Hand Image Editing (NeurIPS 2026 · similarity 0.87)
- TACO: Benchmarking Generalizable Bimanual Tool-ACtion-Object Understanding (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.