Research map / Vision-language and multimodal / Robot action and navigation
VIP: Vision Instructed Pre-training for Robotic Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VIP, a vision-instructed robot manipulation pre-training method that conditions a Transformer policy (VIRT) on current observation, a future/cropped image, and progressively masked sparse point flows instead of text instructions.
Problem
Robotic manipulation policies struggle to use text instructions to specify task targets given current robot data diversity, making large-scale pre-training ambiguous and ineffective.
Approach
Replace text instructions with vision instructions: a future frame plus sparse point flows (progressively masked during pre-training) are used during pre-training, and a cropped image of the target object is used at fine-tuning/inference, with a fully Transformer-based policy (VIRT) initialized from DINOv2 and trained with uncertainty-weighted loss.
Key results
- VIRT with VIP pre-training reaches 0.42 on Pour Blueberries, 0.71 on Open Lid, and 0.37 on Clean Table, versus 0.30/0.65/0.29 without pre-training.
- On simulated tasks VIRT with pre-training achieves 0.92 Move Box, 0.74 Transport Box, 0.68 Stack Boxes; text-instruction inference drops Transport Box to 0.19 and Stack Boxes to 0.06.
- Pre-training with only 10% of data hurts performance (Move Box 0.81 vs 0.87 without pre-training).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VIRT with VIP pre-training instead of ACT with pre-training | VIRT and ACT · Pour Blueberries | success rate: 0.34 -> 0.42 | Helped |
| Use VIRT with VIP pre-training instead of ACT with pre-training | VIRT and ACT · Open Lid | success rate: 0.59 -> 0.71 | Helped |
| Use VIRT with VIP pre-training instead of ACT with pre-training | VIRT and ACT · Clean Table | success rate: 0.30 -> 0.37 | Helped |
| Use VIRT with VIP pre-training instead of ACT with pre-training | VIRT and ACT · Move Box | success rate: 0.90 -> 0.92 | Helped |
| Use VIRT with VIP pre-training instead of ACT with pre-training | VIRT and ACT · Transport Box | success rate: 0.65 -> 0.74 | Helped |
| Use VIRT with VIP pre-training instead of ACT with pre-training | VIRT and ACT · Stack Boxes | success rate: 0.58 -> 0.68 | Helped |
| Add DINOv2 encoder initialization (ablation) | VIRT · Move Box | success rate: 0.80 -> 0.86 | Helped |
| Add uncertainty to the supervision loss (ablation) | VIRT with DINOv2 initialization · Move Box | success rate: 0.86 -> 0.90 | Helped |
| Add random masking of input image pixels (ablation) | VIRT with DINOv2 initialization and uncertainty · Move Box | success rate: 0.90 -> 0.92 | Helped |
| Combine future images and sparse point flows in pre-training instead of using only future images (ablation) | VIRT · Move Box | success rate: 0.87 -> 0.92 | Helped |
| Combine future images and sparse point flows in pre-training instead of using only sparse point flows (ablation) | VIRT · Transport Box | success rate: 0.51 -> 0.74 | Helped |
| Use cropped target images rather than text instructions at inference (ablation) | VIRT with F+S pre-training · Transport Box | success rate: 0.19 -> 0.74 | Helped |
Surprising findings
- Pre-training on a small amount of data (10%) degrades performance relative to no pre-training.
- Text-instructed policies achieve comparable success on the single-box task but fail badly on multi-object tasks (Transport 0.19, Stack 0.06).
Limitations
- Future image from a diffusion world model is unreliable in real scenes, forcing use of cropped object images instead.
- Real-robot evaluation uses only three tasks with 100 trials each and a single robot platform.
- Cropped target image requires an external detector/segmenter (YOLOv10-small) or other specification method.
Experimental setup
Datasets: DROID Isaac Gym teleoperated simulation demos Cobot Magic real-robot demos
Benchmarks: Pour Blueberries Open the Lid Clean the Table Move a Single Box Transport the Specified Box Stack the Specified Boxes
Metrics: success rate inference speed
Baselines: ConvMLP Diffusion Policy ACT text instruction policy
Builds on: DINOv2 CoTracker Diffusion Policy ACT
Closest papers by meaning
- Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasets (ICLR 2025 · similarity 0.89)
- Zero-Shot Robotic Manipulation with Pre-Trained Image-Editing Diffusion Models (ICLR 2024 · similarity 0.89)
- Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation (ICLR 2024 · similarity 0.88)
- Object-Centric Prompt-Driven Vision-Language-Action Model for Robotic Manipulation (CVPR 2025 · similarity 0.88)
- Robotic Visual Instruction (CVPR 2025 · similarity 0.87)
- PriorVLA: Prior-Preserving Adaptation for Vision-Language-Action Models (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.