Research map / Video / Video generation
Show Me What and Tell Me How: Video Synthesis via Multimodal Conditioning
Check your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use bidirectional masked-sequence video prediction instead of ART-V autoregressive prediction | Video generation · Multimodal VoxCeleb | FVD: 60.342 -> 46.763 | Helped |
| Use bidirectional masked-sequence video prediction instead of ART-V autoregressive prediction | Video generation · iPER | FVD: 277.604 -> 209.127 | Helped |
| Combine text and visual controls rather than using text only | MMVID · Shapes | Avg accuracy (%): 98.12 -> 99.24 | Helped |
| Use all four negative video augmentations in the VID task (ablation) | MMVID · Shapes | Avg accuracy (%): 93.34 -> 94.14 | Helped |
| Use color jitter only instead of all four VID augmentations (ablation) | MMVID · Shapes | Color accuracy (%): 88.28 -> 91.02 | Helped |
| Use all four VID augmentations instead of frame shuffling only (ablation) | MMVID · Shapes | Avg accuracy (%): 91.80 -> 94.14 | Helped |
| Swap condition signals for REL negatives instead of sampling annotation-mismatched negatives (ablation) | MMVID · Multimodal VoxCeleb, 128×128, 100K | FVD: 117.128 -> 103.622 | Helped |
| Swap condition signals for REL negatives instead of sampling annotation-mismatched negatives (ablation) | MMVID · Multimodal VoxCeleb, 128×128, 50K | FVD: 109.471 -> 123.147 | Hurt |
| Increase mask-predict beam count from 15 to 20 (ablation) | MMVID · Multimodal VoxCeleb | FVD: 92.828 -> 97.247 | Hurt |
| Enable score-based early stopping (ablation) | MMVID with B=3 · Multimodal VoxCeleb | FVD: 96.288 -> 99.899 | Hurt |
| Add text augmentation (ablation) | MMVID · Multimodal VoxCeleb | Human preference (%): 45.5% -> 54.5% | Helped |
| Add text augmentation (ablation) | MMVID · Multimodal VoxCeleb | Human preference (%): 38.8% -> 61.2% | Helped |
Closest papers by meaning
- Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization (ICML 2024 · similarity 0.89)
- Enhancing Motion in Text-to-Video Generation with Decomposed Encoding and Conditioning (NeurIPS 2024 · similarity 0.89)
- Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video Synthesis (CVPR 2024 · similarity 0.88)
- VideoPoet: A Large Language Model for Zero-Shot Video Generation (ICML 2024 · similarity 0.88)
- Encapsulated Composition of Text-to-Image and Text-to-Video Models for High-Quality Video Synthesis (CVPR 2025 · similarity 0.88)
- VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesis (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.