Research map / 3D vision / 3D generation

MolmoMotion: Forecasting Point Trajectories in 3D with Language Instruction

NeurIPS 2026 · 3D vision › 3D generation · approach: Scene-aware human motion synthesis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces goal-conditioned 3D point trajectory forecasting, a 1.16M-clip auto-annotated dataset (MolmoMotion-1M), a human-verified benchmark (PointMotionBench), and two Molmo2-based predictors (autoregressive and flow-matching) that outperform baselines and transfer to robot planning and trajectory-guided video generation.

Problem

Forecasting future motion of object-attached points is needed for robotics planning and video generation, but existing methods rely on pixels, category-specific 3D templates, or image-plane 2D tracks that entangle camera motion.

Approach

Predict future metric 3D world-frame trajectories of object-attached query points conditioned on RGB history and a language action description, using a VLM backbone with either autoregressive coordinate-text decoding or a DiT flow-matching head, pretrained on a large automatically annotated 3D track corpus.

Key results

• On PointMotionBench, MolmoMotion-AR with 3 frames achieves HOT3D ADE 0.109 m vs 0.129 for the best baseline (ObjectForesight), and WorldTrack ADE 0.143 m vs 0.548 for Robot4DGen.
• On MolmoSpaces Franka pick-and-place, MolmoMotion initialization raises final average success from 56.0% to 76.3% over Molmo2 initialization.
• DaS+MolmoMotion improves over CogVideoX-5B on all five VBench motion metrics and beats Wan2.2-I2V-A14B on four of five.

Surprising findings

• Non-parametric baselines (Static, Extrapolate) are competitive with or stronger than several learned baselines, including pixel-space video prediction methods.
• The autoregressive variant outperforms the flow-matching variant on deterministic metrics, despite flow matching being better suited to uncertainty.

Limitations

• Only 8 query points per example in stage 2 due to Molmo2 context length, so dense object geometry is not captured and multiple forward passes are needed.
• Closed-loop real-robot evaluation is left to future work; downstream evaluation is limited.
• Pixel-space baselines are evaluated via a 3D tracking pipeline, which may disadvantage them; PointMotionBench is small (742 clips).

Experimental setup

Datasets: MolmoMotion-1M EgoDex HD-EPIC Xperience-10M MolmoSpaces DROID YT-VIS Stereo4D HOT3D WorldTrack DAVIS

Benchmarks: PointMotionBench

Metrics: ADE FDE PWT VBench temporal consistency VBench subject consistency VBench motion smoothness VBench dynamic degree VBench background consistency task success rate

Baselines: Static Extrapolate Wan2.2-5B Cosmos Predict ObjectForesight EgoScaler Robot4DGen Track2Act CogVideoX-5B Wan2.2-I2V-A14B

Builds on: Molmo2 MolmoBot Stereo4D ViPE AllTracker SAM 3 DaS

Closest papers by meaning

• PointZero: 3D Point Track Completion for Learning Transferable 3D Dynamics (NeurIPS 2026 · similarity 0.90)
• AutoTraces: Autoregressive Trajectory Forecasting via Multimodal Large Language Models (CVPR 2026 · similarity 0.88)
• D3D-VLP: Dynamic 3D Vision-Language-Planning Model for Embodied Grounding and Navigation (CVPR 2026 · similarity 0.87)
• World Motion Models: Flexible Sequence Modeling of SE(3) Trajectories (NeurIPS 2026 · similarity 0.87)
• FutureHuman3D: Forecasting Complex Long-Term 3D Human Behavior from Video Observations (CVPR 2024 · similarity 0.87)
• PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planning (ICML 2025 · similarity 0.87)