Research map / Video / Video generation
Solving New Tasks by Adapting Internet Video Knowledge
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies adapting a large pretrained text-to-video model (AnimateDiff) to in-domain robotics data via direct finetuning, DreamBooth subject customization, and probabilistic score composition (including a proposed inverse variant), evaluated as visual planners and policy supervisors on MetaWorld and DeepMind Control.
Problem
Pretrained text-to-video models generalize well via text conditioning but may not capture environment-specific visual and dynamics details, while in-domain robot video data is scarce and does not support generalization to novel language-specified tasks.
Approach
Compose scores of a frozen large pretrained video model with a small in-domain video model, and propose an inverse probabilistic adaptation in which the pretrained model controls the sampling while the in-domain model contributes domain-specific information; compare this with direct finetuning and subject customization.
Key results
- Inverse Probabilistic Adaptation reaches 68.3% overall success rate on 9 MetaWorld tasks under policy supervision, versus 39.8% for vanilla AnimateDiff.
- Subject Customization achieves the highest overall Dog/Humanoid walking return of 146.3, versus 103 for vanilla AnimateDiff.
- Probabilistic Adaptation attains 30.4% overall success in visual planning on MetaWorld, versus 22.2% for vanilla AnimateDiff, while Inverse Probabilistic Adaptation retains 27.4% with suboptimal data where the in-domain model drops to 20.0%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use inverse rather than standard probabilistic adaptation | AnimateDiff policy supervision · MetaWorld | success rate (%): 40.2 -> 68.3 | Helped |
| Use inverse rather than standard probabilistic adaptation | AnimateDiff visual planning · MetaWorld | success rate (%): 30.4 -> 25.9 | Mixed |
| Use subject customization from static images | AnimateDiff policy supervision · Continuous control walking | episode return: 103 -> 146.3 | Helped |
| Directly finetune the motion module | AnimateDiff policy supervision · Continuous control walking | episode return: 103 -> 78.1 | Hurt |
| Use inverse probabilistic adaptation with suboptimal demonstrations | AnimateDiff visual planning · MetaWorld | success rate (%): 20.0 -> 27.4 | Helped |
| Use probabilistic adaptation with suboptimal demonstrations | AnimateDiff visual planning · MetaWorld | success rate (%): 20.0 -> 25.2 | Helped |
| Adapt with inverse probabilistic adaptation rather than vanilla AnimateLCM | AnimateLCM policy supervision · MetaWorld | success rate (%): 37.5 -> 65.2 | Helped |
| Adapt with inverse probabilistic adaptation rather than vanilla AnimateLCM | AnimateLCM visual planning · MetaWorld | success rate (%): 17.8 -> 29.6 | Helped |
| Adapt using probabilistic adaptation | AnimateDiff visual planner · iTHOR | success rate (%): 15.6 -> 28.9 | Helped |
| Adapt using inverse probabilistic adaptation | AnimateDiff visual planner · iTHOR | success rate (%): 15.6 -> 23.0 | Helped |
| Use probabilistic adaptation instead of inverse probabilistic adaptation | AnimateDiff free-form generation · MetaWorld seen tasks | FVD: 928.6 -> 848.3 | Helped |
| Use direct finetuning instead of inverse probabilistic adaptation | AnimateDiff free-form generation · MetaWorld unseen tasks | FVD: 1250.3 -> 915.0 | Helped |
Surprising findings
- Subject Customization, which uses only static images and no video demonstrations, substantially improves continuous locomotion policy performance over the base pretrained video model.
- Direct finetuning achieves the highest FVD on unseen tasks yet does not translate into higher visual-planning success.
Limitations
- Performance differences between the forward and inverse adaptation directions are hard to explain across policy supervision and visual planning setups.
- Results rely on a small set of tasks and seeds with high variance across rollouts.
- Suboptimal-data study is limited to one suboptimal policy (30% expert actions) and a single setting.
Experimental setup
Datasets: MetaWorld-v2 demonstrations DeepMind Control Suite demonstrations iTHOR (AVDC dataset) WebVid-10M
Benchmarks: MetaWorld-v2 DeepMind Control Suite (Dog, Humanoid walking) iTHOR
Metrics: success rate episode return FVD
Baselines: Vanilla AnimateDiff In-Domain-Only Direct Finetuning Subject Customization Probabilistic Adaptation AnimateLCM
Builds on: AnimateDiff DreamBooth Video-TADPoLe AVDC UniPi TD-MPC
Closest papers by meaning
- RoboDreamer: Learning Compositional World Models for Robot Imagination (ICML 2024 · similarity 0.89)
- VideoWorld: Exploring Knowledge Learning from Unlabeled Videos (CVPR 2025 · similarity 0.89)
- PriorVLA: Prior-Preserving Adaptation for Vision-Language-Action Models (NeurIPS 2026 · similarity 0.89)
- Mitigating the Human-Robot Domain Discrepancy in Visual Pre-training for Robotic Manipulation (CVPR 2025 · similarity 0.89)
- VideoVLA: Video Generators Can Be Generalizable Robot Manipulators (NeurIPS 2025 · similarity 0.89)
- RoboAlign-R1: Distilled Multimodal Reward Alignment for Robot Video World Models (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.