Research map / Video / Video segmentation and tracking
Object-centric 3D Motion Field for Robot Learning from Human Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes learning object-centric dense 3D motion fields from noisy RGBD human videos, using a simulation-trained denoising motion-field estimator and an image-based policy (Gaussian or diffusion UNet) for zero-shot robot control with a gripper.
Problem
Extracting accurate action representations from noisy RGBD human videos for zero-shot robot policy learning; existing representations (video frames, pixel flow, point-cloud flow, SE(3) poses) lose information, are noisy, or are limited to rigid bodies.
Approach
Represent actions as a dense object-centric 3D motion field estimated by a UNet trained on simulated noisy depth and pixel-flow data (with intrinsics maps and noise augmentations), then train an image-to-motion-field policy on human videos and convert predicted fields to SE(3) actions via Kabsch+RANSAC.
Key results
- Reduces 3D motion estimation error by over 50% compared to the direct 3D flow baseline on real-world test videos.
- Achieves about 55% average zero-shot success rate across diverse tasks where prior approaches reach roughly 10% or less.
- Policies trained solely on human hand videos perform fine-grained insertion with 2.5mm tolerance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a denoising 3D motion field estimator instead of direct depth-and-tracker lifting | 3D motion estimation · Real-world object motion videos | 3D motion estimation error: over 50\% | Helped |
| Use object-centric 3D motion fields predicted from human videos | Robot control · Diverse real-world manipulation tasks | average success rate: 55\% vs. \lesssim 10\% | Helped |
| Add coordinate maps and inverse focal length to the estimator input (ablation) | 3D motion field estimator · Real-world motion prediction | motion prediction accuracy | Helped |
| Use focal length values in the intrinsic map (ablation) | 3D motion field estimator · Motion prediction across a relatively small FoV variation | motion prediction accuracy | Helped |
| Use a diffusion policy instead of a Gaussian policy head (ablation) | Robot policy · Fine-grained manipulation tasks | task success | Helped |
| Mask unnecessary regions during the diffusion reverse step (ablation) | Diffusion policy · Robot policy training and performance | training efficiency and task performance | Helped |
| Apply object masking augmentation during training (ablation) | Robot policy · Human-to-robot transfer | task performance | Helped |
Surprising findings
- Focal length/intrinsics maps are critical for motion prediction even for small FoV variations of about 10 degrees.
Limitations
- Does not handle fully occluded object segments in human videos.
- Assumes a fixed camera and only parallel-jaw grippers rather than robotic hands.
- Soft-body objects such as cloth are not addressed.
Experimental setup
Datasets: ShapeNet simulated randomly generated rigid bodies real-world human object interaction RGBD videos collected with Intel D435
Benchmarks: real-world 3D motion estimation test set real-world robot tasks: pick rotate place, line tracking, pushing, wrench, insertion
Metrics: MSE of object translation Frobenius norm of rotation error task success rate
Baselines: General Flow direct 3D flow method UniPi ATM Track2Act Im2Flow2Act SPOT TI
Builds on: UNet CoTracker3 SAM2 General Flow DDIM
Closest papers by meaning
- Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasets (ICLR 2025 · similarity 0.88)
- PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planning (ICML 2025 · similarity 0.88)
- VidBot: Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulation (CVPR 2025 · similarity 0.87)
- Pre-training Auto-regressive Robotic Models with 4D Representations (ICML 2025 · similarity 0.87)
- Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations (ICML 2025 · similarity 0.87)
- Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Training (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.