Research map / Point clouds / Point cloud classification

PointZero: 3D Point Track Completion for Learning Transferable 3D Dynamics

NeurIPS 2026 · Point clouds › Point cloud classification · approach: Point-language and open-vocabulary 3D · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes 3D point track completion, a robot-free pre-training objective that predicts dense future 3D point tracks from one RGB-D image and sparse partial tracks, trained on a 2.9M-frame synthetic dataset, and post-trained for action-conditioned dynamics and imitation learning.

Problem

Learning transferable 3D dynamics priors typically requires robot action labels or scene-specific interaction data, which excludes web video and limits scalability.

Approach

Pre-train a Perceiver-IO-conditioned diffusion transformer to complete 3D point tracks from one RGB-D view and a few partial tracks, using only track supervision (no action labels), then post-train by swapping conditioning to end-effector pose or adding an action head.

Key results

• On the PGND benchmark, PointZero-FT achieves best MDE on 4 of 6 scenes after post-training, outperforming PGND (e.g., Bread MDE 1.5 vs 2.0 cm).
• On synthetic held-out data, PointZero-JiT-oracle-10 reduces deformable MDE to 2.85 cm versus 9.20 cm for PTv3.
• Imitation learning: PointZero reaches 99.8% on Blockstack, 93.1% on Microwave, and 100% on real-world Drawer and Cup, highest or joint-highest on 6 of 7 tasks.

Surprising findings

• JiT does not consistently outperform flow matching when both are trained sufficiently long.
• Pre-trained PointZero is competitive with scene-specific baselines in zero-shot PGND evaluation despite never interacting with the objects.

Limitations

• Synthetic pre-training data does not capture full real-world material diversity, hand-object interactions, clutter, or long horizons.
• Relatively simple conditioning signals (partial point tracks, end-effector pose).
• Downstream evaluations are task-specific and small-scale.

Experimental setup

Datasets: PointZero synthetic dataset PGND benchmark

Benchmarks: PointZero synthetic held-out set PointZero real-world dataset PGND benchmark PGND zero-shot 3PoinTr imitation learning tasks

Metrics: MDE MSE Chamfer Distance EMD imitation success rate

Baselines: GBND ParticleFormer PGND PTv3 DP3 Diffusion Policy 3PoinTr ATM PointWorld

Builds on: Perceiver-IO JiT DINOv2 PGND 3PoinTr

Closest papers by meaning

• MolmoMotion: Forecasting Point Trajectories in 3D with Language Instruction (NeurIPS 2026 · similarity 0.90)
• Lifting Motion to the 3D World via 2D Diffusion (CVPR 2025 · similarity 0.88)
• PointMapPolicy: Structured Point Cloud Processing for Multi-Modal Imitation Learning (NeurIPS 2025 · similarity 0.87)
• VidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulation (NeurIPS 2024 · similarity 0.87)
• Pre-training Auto-regressive Robotic Models with 4D Representations (ICML 2025 · similarity 0.87)
• World Motion Models: Flexible Sequence Modeling of SE(3) Trajectories (NeurIPS 2026 · similarity 0.87)