Research map / Video / Action recognition
Action Images: End-to-End Policy Learning via Multiview Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Action Images renders 7-DoF robot actions as pixel-grounded multi-view action videos and fine-tunes Wan2.2 to jointly generate robot and action videos, yielding a zero-shot policy decoded back to continuous control.
Problem
Video-based world models do not translate their predictive ability into policy generalization because actions are handled by separate heads or non-spatially-grounded representations, limiting transfer across viewpoints and environments.
Approach
Encode each 7-DoF action as three semantic 3D points projected into each camera view and rendered as RGB Gaussian heatmaps (action images), so actions live in the same video space as observations; a pretrained video backbone then jointly generates robot and action videos and serves as a zero-shot policy, with actions decoded via multi-view ray casting.
Key results
- Zero-shot RLBench success on held-out tasks reaches 60% on reach target and 50% on close drawer, and the method is best on all 4 RLBench and all 5 real-world zero-shot tasks in the table (e.g., Place Cup 40%).
- Joint generation video quality: PSNR 23.48 and SSIM 78.62% vs. TesserAct 20.83 / 59.20%, with 3D action error 12.2 vs. 19.0.
- Video-to-action labeling Jaccard avg 46.71 vs. CoTracker 31.20 and TAPIR 29.77; action-conditioned video PSNR 31.35 vs. Tora 19.76.
Surprising findings
- Without an action head, the in-domain RLBench average (20.6) only matches TesserAct and is below Cosmos-Policy (20.0)-level baselines on several tasks, with 0% on open door and phone base, indicating the zero-shot decoder is weaker than the optional head.
- Adding the optional learned action head raises the RLBench in-domain average from 20.6 to 36.7.
Limitations
- Only open-loop evaluation; not yet a closed-loop policy.
- Real-world success rates remain low (0-45%) on several tasks.
- Requires multi-view camera calibration/intrinsics and extrinsics, estimated with VGGT for in-the-wild images.
Experimental setup
Datasets: RLBench DROID BridgeV2 Robot-Colosseum
Benchmarks: RLBench Real-world xArm tasks Bridge DROID FR3M
Metrics: task success rate PSNR SSIM FVD LPIPS 2D trajectory error 3D trajectory error Jaccard
Baselines: MV-Policy π0.5 MolmoAct TesserAct Cosmos-Policy Cosmos-Predict2.5-14B Tora TAPIR CoTracker3
Builds on: Wan 2.2 ReCamMaster Plücker embedding flow matching VGGT
Closest papers by meaning
- EA-WM: Event-Aware Generative World Model with Structured Kinematic-to-Visual Action Fields (NeurIPS 2026 · similarity 0.91)
- Learning to Act from Actionless Videos through Dense Correspondences (ICLR 2024 · similarity 0.90)
- Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Training (NeurIPS 2024 · similarity 0.90)
- DiT4DiT: Jointly Modeling Video Dynamics and Actions for Generalizable Robot Control (NeurIPS 2026 · similarity 0.90)
- World Action Verifier: Self-Improving World Models via Forward-Inverse Asymmetry (NeurIPS 2026 · similarity 0.89)
- SyncWorld: Visual Calibration Enables World Models as Zero-Shot Simulators (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.