Research map / Vision-language and multimodal / Robot action and navigation

PIVOT-R: Primitive-Driven Waypoint-Aware World Model for Robotic Manipulation

NeurIPS 2024 · Vision-language and multimodal › Robot action and navigation · approach: Memory and subgoal-based manipulation · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PIVOT-R, a primitive-driven waypoint-aware world model that uses a VLM to parse primitive actions and predicts waypoint features, plus an asynchronous hierarchical executor running modules at different frequencies, evaluated on the SeaWave simulation benchmark.

Problem

Language-guided robotic manipulation maps instructions and observations directly to low-level actions, neglecting key waypoints and world dynamics, and sequential execution of VLM and world model modules is computationally redundant.

Approach

Predict only task-relevant waypoint features (conditioned on VLM-parsed primitive actions) with a scene prediction transformer, then decode low-level actions from those features, and run the VLM, scene predictor, and action head asynchronously at slow-to-fast frequencies.

Key results

• PIVOT-R reaches a mean success rate of 74.19% on SeaWave across four levels, 19.45% above the best baseline Surfer (54.74%).
• The asynchronous hierarchical executor gives about 28x faster execution (27 ms vs 756 ms) with a 2.9% performance drop relative to the synchronous variant.
• On the SIMPLER benchmark PIVOT-R averages 0.393 versus 0.295 for the best Octo variant.

Measured effects reported in the paper

Surprising findings

• Pixel-level prediction of waypoint images performed worse than feature-level prediction (66.32 vs 74.19 mean).
• Co-training with Ego4D human video reduced seen-scenario performance, while pre-training helped in unseen backgrounds and with distractors.

Limitations

• Action execution is sometimes inconsistent with instructions (e.g., 'push left' executed as 'push front').
• Evaluation is mainly in a single simulation benchmark (SeaWave) with a small real-world study.

Experimental setup

Datasets: SeaWave Ego4D SIMPLER (evaluation)

Benchmarks: SeaWave SIMPLER real-world Franka tasks

Metrics: success rate inference time (ms)

Baselines: Gato BC-Z Octo SUSIE RT-1 GR-1 Surfer

Builds on: RT-1 I-JEPA CLIP LLaVA Surfer

Closest papers by meaning

• SAM2Act: Integrating Visual Foundation Model with A Memory Architecture for Robotic Manipulation (ICML 2025 · similarity 0.89)
• RoboCerebra: A Large-scale Benchmark for Long-horizon Robotic Manipulation Evaluation (NeurIPS 2025 · similarity 0.89)
• Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation (ICLR 2025 · similarity 0.89)
• Chain-of-Action: Trajectory Autoregressive Modeling for Robotic Manipulation (NeurIPS 2025 · similarity 0.89)
• RoboProcessBench: Benchmarking Process-Aware Understanding in Visual-Language Robotic Manipulation (NeurIPS 2026 · similarity 0.89)
• Robotic Visual Instruction (CVPR 2025 · similarity 0.89)