Research map / Video / Video generation
Video-As-Prompt: Unified Semantic Control for Video Generation
Yuxuan Bian, Xin Chen, Zenan Li, Tiancheng Zhi, Shen Sang, Linjie Luo, Qiang Xu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use mixture-of-transformers (frozen DiT + parallel expert with full attention) instead of single-branch full finetuning (ablation) | VAP on CogVideoX-I2V-5B · VAP-Data test | Semantic alignment score: 68.74 -> 70.44 | Improved |
| use MoT instead of single-branch LoRA finetuning (ablation) | VAP on CogVideoX-I2V-5B · VAP-Data test | Dynamic degree: 72.92 -> 77.08 dynamic degree; alignment 69.08 -> 70.44 | Mixed |
| use bidirectional full attention instead of unidirectional cross-attention injection (ablation) | VAP on CogVideoX-I2V-5B · VAP-Data test | Semantic alignment score: 67.16 -> 70.44 | Improved |
| use MoT full attention instead of residual addition injection (ablation) | VAP on CogVideoX-I2V-5B · VAP-Data test | Semantic alignment score: 55.99 -> 70.44 | Improved |
| add temporally biased RoPE instead of identical RoPE for reference and target (ablation) | VAP on CogVideoX-I2V-5B · VAP-Data test | Semantic alignment score: 68.98 -> 70.44 | Improved |
| add a width bias (reference placed left) in addition to temporal bias (ablation) | VAP on CogVideoX-I2V-5B · VAP-Data test | Semantic alignment score: 70.44 -> 69.05 | Hurt |
| scale training pairs from 1K to 100K (ablation) | VAP on CogVideoX-I2V-5B · VAP-Data test | Semantic alignment score: 63.91 (1K) -> 66.28 (10K) -> 68.23 (50K) -> 70.44 (100K) | Improved |
| swap backbone to Wan2.1-I2V-14B with expert in 1/4 of layers (ablation) | VAP · VAP-Data test | Dynamic degree: dynamic degree 77.08 -> 79.17; alignment 70.44 -> 70.23 | Mixed |
| place MoT expert in all layers instead of only odd layers (ablation) | VAP on CogVideoX-I2V-5B · VAP-Data test | Semantic alignment score: 70.22 -> 70.44 | Improved |
| add noise to the reference video prompt (ablation) | VAP on CogVideoX-I2V-5B · VAP-Data test | Semantic alignment score: 70.44 -> 70.18 | Hurt |
| use more reference videos (1-3) as prompts | VAP · multi-reference test | qualitative | No clear effect |
| use structure-controlled VACE with raw video condition instead of depth/optical flow | VACE · VAP-Data test | Semantic alignment score: 46.71 (flow) -> 43.35 (depth) -> 35.38 (original) | Hurt |
Closest papers by meaning
- DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generation (CVPR 2025 · similarity 0.89)
- VideoBooth: Diffusion-based Video Generation with Image Prompts (CVPR 2024 · similarity 0.88)
- Motion Prompting: Controlling Video Generation with Motion Trajectories (CVPR 2025 · similarity 0.87)
- VSTAR: Generative Temporal Nursing for Longer Dynamic Video Synthesis (ICLR 2025 · similarity 0.87)
- VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control (ICLR 2025 · similarity 0.87)
- Generative Video Diffusion for Unseen Novel Semantic Video Moment Retrieval (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
