Research map / Video / Video generation
Precise Action-to-Video Generation Through Visual Action Prompts
Yuang Wang, Chao Wen, Haoyu Guo, Sida Peng, Minghan Qin, Hujun Bao, Xiaowei Zhou, Ruizhen Hu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use skeleton visual action prompts instead of raw 7-DoF state | CogVideoX video model · RT-1 | ST-IoU: 0.507 -> 0.604 | Improved |
| use skeleton prompts instead of text action descriptions | CogVideoX video model · DROID | ST-IoU: 0.239 -> 0.450 | Improved |
| use skeleton prompts instead of raw state | CogVideoX video model · DROID novel lab | J&F: 33.4 -> 52.5 | Improved |
| joint training on RT-1, DROID and EgoVid with skeleton prompts | Skeleton (Ours) · DROID | FVD: 141.8 -> 124.4 | Improved |
| joint training on RT-1, DROID and EgoVid with skeleton prompts | Skeleton (Ours) · RT-1 | PSNR: 25.98 -> 24.90 | Hurt |
| joint training with skeleton prompts on DROID subsets | Skeleton (Ours) · DROID known lab and skill | J&F: 53.6 -> 60.0 | Improved |
| use mesh rendering or depth map prompts instead of skeleton (ablation) | Visual action prompt form · DROID | ST-IoU: 0.450 -> 0.586 (mesh) | Improved |
| add ControlNet branch for injecting action prompts (ablation) | Visual action prompt model · DROID | ST-IoU: 0.396 -> 0.450 | Improved |
| add main-branch injection with LoRA fine-tuning of DiT (ablation) | Visual action prompt model · DROID | ST-IoU: 0.442 -> 0.450 | Improved |
Closest papers by meaning
- Generation-for-Understanding with Structured Action Scripts for Embodied Multimodal Learning (NeurIPS 2026 · similarity 0.89)
- DiT4DiT: Jointly Modeling Video Dynamics and Actions for Generalizable Robot Control (NeurIPS 2026 · similarity 0.88)
- Motion Prompting: Controlling Video Generation with Motion Trajectories (CVPR 2025 · similarity 0.88)
- Beyond World-Frame Action Heads: Motion-Centric Action Frames for Vision-Language-Action Models (NeurIPS 2026 · similarity 0.87)
- Object-Centric Prompt-Driven Vision-Language-Action Model for Robotic Manipulation (CVPR 2025 · similarity 0.87)
- Grounding Video Models to Actions through Goal Conditioned Exploration (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
