Research map / 3D vision / Human pose estimation
What's the Move? Hybrid Imitation Learning via Salient Points
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SPHINX, a hybrid imitation learning policy that predicts salient points and waypoints from point clouds for long-range motion and switches to a diffusion-based dense policy on wrist images for precise phases.
Problem
Visuomotor imitation learning policies struggle to generalize across visual and spatial variations and to perform precise, long-horizon manipulation from limited demonstrations.
Approach
A hybrid policy that uses point clouds to predict per-point salient-point probabilities and offsets to form waypoints for long-range motion, and switches to a wrist-image diffusion policy for dense precise actions near salient points; demonstrations are collected through a web GUI that records salient points and waypoints alongside teleoperated dense actions.
Key results
- SPHINX achieves 86.7% average success across 4 real-world and 2 simulated tasks, outperforming the next best IL baseline by 41.1% on average over 440 real-world trials.
- On Robomimic waypoint ablations, SPHINX with temporal augmentation reaches 93.5% on Can and 86.5% on Square, versus 70% and 13% for vanilla waypoint.
- SPHINX completes Cup Stack in 9.2 s on average versus 15.6 s for diffusion policy, a 1.7x speedup with a faster controller at test time.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SPHINX instead of Diffusion Policy | Drawer policy · Drawer — unseen height | success: 4/10 -> 9/10 | Helped |
| Use SPHINX instead of Diffusion Policy | Drawer policy · Drawer — novel viewpoint | success: 1/10 -> 9/10 | Helped |
| Use SPHINX instead of Diffusion Policy | Cup Stack policy · Cup Stack — visual distraction | success: 1/10 -> 8/10 | Helped |
| Use SPHINX instead of Diffusion Policy | Cup Stack policy · Cup Stack — novel viewpoint | success: 0/10 -> 9/10 | Helped |
| Use a 2× sped-up waypoint controller | SPHINX on Cup Stack · Cup Stack | completion time: 15.6 seconds -> 9.2 seconds; 1.7× speedup | Helped |
| Add temporal augmentation (ablation) | Vanilla Waypoint · Can | success: 68% -> 70% | Helped |
| Add temporal augmentation (ablation) | Vanilla Waypoint + Auxiliary Salient Points · Can | success: 70% -> 78% | Helped |
| Add temporal augmentation (ablation) | SPHINX · Can | success: 77.5% -> 93.5% | Helped |
| Add temporal augmentation (ablation) | Vanilla Waypoint · Square | success: 8% -> 13% | Helped |
| Add temporal augmentation (ablation) | Vanilla Waypoint + Auxiliary Salient Points · Square | success: 68% -> 65.5% | Hurt |
| Add temporal augmentation (ablation) | SPHINX · Square | success: 75.5% -> 86.5% | Helped |
| Add salient point prediction as an auxiliary task to Vanilla Waypoint (ablation) | Vanilla Waypoint · Square | success | Helped |
Surprising findings
- Vanilla waypoint prediction is notably worse than diffusion policy on the Square and real Drawer tasks, showing shorter prediction horizons alone do not help without salient point prediction.
- Fine-tuned OpenVLA performs poorly on the tasks, which the authors attribute to single third-person view input and embodiment gap from Bridge pretraining data.
Limitations
- Dense policy is sometimes imprecise for grasping or last-mile insertion, the main failure mode.
- The linear waypoint controller limits the method to quasistatic tasks without fast dynamic motion.
- Waypoint policy performance is bounded by point cloud quality and multi-view calibration error.
Experimental setup
Datasets: Robomimic Can Robomimic Square self-collected real-world demonstrations (Drawer, Cup Stack, Coffee Making, Train Track)
Benchmarks: Robomimic Can Robomimic Square Drawer (real) Cup Stack (real) Coffee Making (real) Train Track (real)
Metrics: success rate average completion time (s)
Baselines: Diffusion Policy HYDRA Fine-tuned OpenVLA 3D Diffusion Policy Vanilla Waypoint Vanilla Waypoint + Auxiliary Salient Points
Builds on: Diffusion Policy HYDRA 3D Diffusion Policy OpenVLA Robomimic
Closest papers by meaning
- GravMAD: Grounded Spatial Value Maps Guided Action Diffusion for Generalized 3D Manipulation (ICLR 2025 · similarity 0.89)
- OSVI-WM: One-Shot Visual Imitation for Unseen Tasks using World-Model-Guided Trajectory Generation (NeurIPS 2025 · similarity 0.88)
- Data Scaling Laws in Imitation Learning for Robotic Manipulation (ICLR 2025 · similarity 0.88)
- Stem-OB: Generalizable Visual Imitation Learning with Stem-Like Convergent Observation through Diffusion Inversion (ICLR 2025 · similarity 0.88)
- Latent Diffusion Planning for Imitation Learning (ICML 2025 · similarity 0.88)
- Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasets (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.