Research map / Video / Video segmentation and tracking
Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| BC fine-tune VPT foundation model on earlygame_keyword | VPT foundation model · Minecraft early-game crafting | relative crafting tables: 2.5x more crafting tables, 6.1x more planks, 4.3x more logs | Helped |
| BC fine-tune VPT foundation model on contractor_house | VPT foundation model · Minecraft early-game crafting | relative crafting tables: 213x more crafting tables, 59x more planks | Helped |
| add early-game BC fine-tuning before RL (ablation) | VPT RL fine-tuning · Minecraft diamond pickaxe reward | reward: 13 -> 25 | Helped |
| add KL loss to pretrained policy during RL (ablation) | RL from early-game model · Minecraft diamond pickaxe reward | reward | Helped |
| initialize RL from VPT foundation model instead of random init (ablation) | RL on diamond pickaxe task · Minecraft diamond pickaxe reward | reward: almost none -> near 13 | Helped |
| train foundation model on IDM-labeled web data instead of contractor data only (ablation) | VPT foundation model · Minecraft zero-shot crafting | crafting rate | Helped |
| train IDM on more contractor data beyond 100 hours (ablation) | IDM labeling for BC · earlygame_keyword zero-shot BC | crafting rate | No clear effect |
| use IDM (non-causal) instead of BC on same contractor data | contractor data use · data efficiency | accuracy vs data | Helped |
| longer foundation model training | VPT foundation model · zero-shot rollout stats | game statistics | No clear effect |
Closest papers by meaning
- Videos are Sample-Efficient Supervisions: Behavior Cloning from Videos via Latent Representations (NeurIPS 2025 · similarity 0.90)
- Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration (ICML 2025 · similarity 0.89)
- Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Training (NeurIPS 2024 · similarity 0.88)
- Towards Principled Representation Learning from Videos for Reinforcement Learning (ICLR 2024 · similarity 0.88)
- Efficient Active Imitation Learning with Random Network Distillation (ICLR 2025 · similarity 0.88)
- GROOT: Learning to Follow Instructions by Watching Gameplay Videos (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.