Research map / Video / Action recognition
Learning to Act from Actionless Videos through Dense Correspondences
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AVDC, which synthesizes task videos with a text-conditioned diffusion model, infers object or camera SE(3) motion from optical flow and the first-frame depth, and converts it into robot actions without any action labels.
Problem
Learning robot policies from video demonstrations without action labels, while keeping video policy training computationally affordable and generalizing across robots and environments.
Approach
Formulate policy learning as text-conditioned video generation, then recover actions from dense optical-flow correspondences between synthesized frames by solving for rigid 3D transforms using the initial depth map, avoiding any action labels or learned inverse-dynamics model.
Key results
- On Meta-World (11 tasks), AVDC (Full) reaches 43.1% overall success vs 16.2% for BC-Scratch (which uses ground-truth actions) and 6.1% for UniPi with replanning.
- On iTHOR object navigation, AVDC achieves 31.3% success while both BC baselines fail entirely.
- Zero-shot transfer from actionless human pushing videos to simulated Visual Pusher reaches 90% success over 40 runs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use AVDC (Full) instead of the strongest listed behavioral cloning baseline | Meta-World · Meta-World overall | success rate: 16.2% -> 43.1% | Helped |
| replace video synthesis followed by flow prediction with direct optical-flow prediction (ablation) | AVDC · Meta-World overall | success rate: 43.1% -> 13.7% | Hurt |
| remove closed-loop replanning (ablation) | AVDC · Meta-World overall | success rate: 19.6% -> 43.1% | Hurt |
| use an inverse-dynamics model to infer actions from synthesized videos (ablation) | AVDC · Meta-World overall | success rate: 6.1% -> 43.1% | Hurt |
| use 10-step DDIM instead of 100 denoising steps (ablation) | AVDC · Meta-World overall | success rate: 43.1% -> 37.5% | Hurt |
| use Language Segment-Anything object masks instead of ground-truth masks (ablation) | AVDC · Meta-World overall | success rate: 43.1% -> 34.5% | Hurt |
| replace RGB-channel-wise first-frame concatenation with frame-wise concatenation (ablation) | AVDC video generation · Bridge video generation | pixel-level MSE | Hurt |
| replace the CLIP-text encoder with T5-base (ablation) | AVDC video generation · video generation | pixel-level MSE | No clear effect |
| increase the maximum number of replanning trials (ablation) | AVDC · Meta-World | success rate | Helped |
| use AVDC for zero-shot robot execution from actionless human pushing videos | AVDC · Visual Pusher | zero-shot success rate: 90% | Helped |
Surprising findings
- The action-label-free AVDC outperforms behavioral cloning baselines that have access to expert actions on Meta-World.
- Directly training diffusion models to predict flow underperforms the two-stage synthesize-then-estimate-flow pipeline (AVDC Flow overall 13.7%).
Limitations
- Optical flow errors on small objects and occlusion by the robot arm cause tracking loss, and real-world performance was low (8 of 10 trials failed).
- Grasp and contact point prediction is heuristic (centroid of mask, top-grasp assumption) and not transferable from human videos.
- Force information is unavailable from RGB videos.
Experimental setup
Datasets: Meta-World iTHOR Visual Pusher human videos Bridge in-house human tabletop videos
Benchmarks: Meta-World iTHOR Visual Pusher Franka Emika Panda real-world tasks
Metrics: success rate MSE of synthesized last frame
Baselines: BC-Scratch BC-R3M UniPi (With Replan) AVDC (Flow) AVDC (No Replan) AVDC (ID)
Builds on: UniPi GMFlow CLIP Factorized spatial-temporal convolution (Make-A-Video style) DDIM
Closest papers by meaning
- Action Images: End-to-End Policy Learning via Multiview Video Generation (NeurIPS 2026 · similarity 0.90)
- Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Training (NeurIPS 2024 · similarity 0.90)
- DiT4DiT: Jointly Modeling Video Dynamics and Actions for Generalizable Robot Control (NeurIPS 2026 · similarity 0.89)
- Grounding Video Models to Actions through Goal Conditioned Exploration (ICLR 2025 · similarity 0.88)
- Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation (ICLR 2025 · similarity 0.88)
- Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasets (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.