Research map / Video / Video generation
TASTE-Rob: Advancing Video Generation of Task-Oriented Hand-Object Interaction for Generalizable Robotic Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces TASTE-Rob, a 100,856-clip static-camera ego-centric hand-object interaction video dataset, and a three-stage pose-refinement pipeline (fine-tuned DynamiCrafter, hand-pose MDM, pose-conditioned adapter) for generating task-oriented HOI videos used as demonstrations for robot imitation learning.
Problem
Existing ego-centric HOI datasets (e.g., Ego4D) have inconsistent camera views and misaligned language-action pairs, and general video diffusion models produce task-oriented hand-object interaction videos with inconsistent grasp poses, limiting their use as imitation learning demonstrations.
Approach
Collect a large, fixed-camera, single-action ego-centric HOI dataset aligned with instructions, fine-tune an I2V diffusion model on it for coarse video generation, then refine the extracted hand-pose sequence with a motion diffusion model conditioned on the environment image and regenerate video conditioned on the refined poses.
Key results
- TASTE-Rob achieves FVD 9.43, KVD 0.03, PIC 0.90 on TASTE-Rob-Test, versus best baseline CogVideoX at FVD 48.72, KVD 0.16, PIC 0.85.
- Pose refinement reduces GTCE from 67.8% (Coarse-TASTE-Rob) to 9.7% and HMDA from 26.4° to 11.3°.
- Robot manipulation success rate in Mujoco rises from 84% (coarse) to 96% with pose refinement.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune DynamiCrafter on TASTE-Rob rather than use general I2V baselines | TASTE-Rob · TASTE-Rob-Test | KVD: 0.16 -> 0.03 | Helped |
| fine-tune DynamiCrafter on TASTE-Rob rather than use general I2V baselines | TASTE-Rob · TASTE-Rob-Test | FVD: 48.72 -> 9.43 | Helped |
| fine-tune DynamiCrafter on TASTE-Rob rather than use general I2V baselines | TASTE-Rob · TASTE-Rob-Test | PIC: 0.85 -> 0.90 | Helped |
| train on TASTE-Rob instead of Ego4D (ablation) | DynamiCrafter · TASTE-Rob-Test | KVD: 0.18 -> 0.04 | Helped |
| train on TASTE-Rob instead of Ego4D (ablation) | DynamiCrafter · TASTE-Rob-Test | FVD: 52.17 -> 10.85 | Helped |
| train on TASTE-Rob instead of Ego4D (ablation) | DynamiCrafter · TASTE-Rob-Test | PIC: 0.77 -> 0.88 | Helped |
| add the three-stage pose-refinement pipeline (ablation) | Coarse-TASTE-Rob · TASTE-Rob-Test | GPV: 0.28 -> 0.24 | Helped |
| add the three-stage pose-refinement pipeline (ablation) | Coarse-TASTE-Rob · TASTE-Rob-Test | GTCE: 67.8% -> 9.7% | Helped |
| add the three-stage pose-refinement pipeline (ablation) | Coarse-TASTE-Rob · TASTE-Rob-Test | HMDA: 26.4° -> 11.3° | Helped |
| add the three-stage pose-refinement pipeline (ablation) | Coarse-TASTE-Rob demonstrations · Mujoco simulation dataset | success rate: 84% -> 96% | Helped |
| add the three-stage pose-refinement pipeline (ablation) | Coarse-TASTE-Rob · TASTE-Rob-Test | FVD: 10.85 -> 9.43 | Helped |
| vary pose-refinement denoising steps (Nrv) (ablation) | TASTE-Rob pose-refinement pipeline · TASTE-Rob-Test | video generation quality | Mixed |
Surprising findings
- General off-the-shelf I2V video diffusion models fail at task-oriented manipulation videos despite generating realistic human videos.
- Ego4D-finetuned generator produced videos too low-quality for pose extraction, so the pose-refinement ablation was only run on TASTE-Rob.
Limitations
- Generation quality degrades when objects undergo significant transformations such as rotation or opening.
- Evaluation on a relatively small simulation set (50 tasks) and a held-out split of the same dataset.
Experimental setup
Datasets: TASTE-Rob Ego4D
Benchmarks: TASTE-Rob-Test Mujoco simulation dataset (Im2Flow2Act platform, 50 tasks)
Metrics: FVD KVD PIC GPV GTCE HMDA success rate
Baselines: DynamiCrafter consistI2V Open-Sora Plan CogVideoX Ego4D-Gen
Builds on: DynamiCrafter MDM (Motion Diffusion Model) ToonCrafter Im2Flow2Act
Closest papers by meaning
- ManiVideo: Generating Hand-Object Manipulation Video with Dexterous and Generalizable Grasping (CVPR 2025 · similarity 0.86)
- HandEdit: A Unified Benchmark for Egocentric Human-to-Robot Dexterous Hand Image Editing (NeurIPS 2026 · similarity 0.84)
- TACO: Benchmarking Generalizable Bimanual Tool-ACtion-Object Understanding (CVPR 2024 · similarity 0.83)
- Object-Centric Prompt-Driven Vision-Language-Action Model for Robotic Manipulation (CVPR 2025 · similarity 0.83)
- Scaffolding Dexterous Manipulation with Vision-Language Models (NeurIPS 2025 · similarity 0.83)
- ComPose: When to Trust Hands for Object Pose Tracking (NeurIPS 2026 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.