Research map / Video / Video generation
MusicInfuser: Making Video Diffusion Listen and Dance
Susung Hong, Ira Kemelmacher-Shlizerman, Brian Curless, Steven M. Seitz
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| select cross-attention layers by layer adaptability instead of all layers (ablation) | MusicInfuser · Qwen3-Omni judge | Overall: 7.80 -> 8.14 | Improved |
| select layers by adaptability instead of evenly distributed layers (ablation) | MusicInfuser · Qwen3-Omni judge | Overall: 7.62 -> 8.14 | Improved |
| select layers by adaptability instead of first layers (ablation) | MusicInfuser · Qwen3-Omni judge | Overall: 7.99 -> 8.14 | Improved |
| use zero-initialized cross-attention (ZICA) instead of direct feature addition (ablation) | MusicInfuser · Qwen3-Omni judge | Overall: 7.92 -> 8.14 | Improved |
| zero-initialize cross-attention output projection (ablation) | MusicInfuser · Qwen3-Omni judge | Video Quality Average: 7.82 -> 8.02 | Improved |
| add Beta-Uniform noise scheduling (ablation) | MusicInfuser · Qwen3-Omni judge | Overall: 7.89 -> 8.14 | Improved |
| add in-the-wild dance data to AIST training (ablation) | MusicInfuser · Qwen3-Omni judge | Overall: 7.95 -> 8.14 | Improved |
| add in-the-wild dance data to AIST training (ablation) | MusicInfuser · Prompt alignment | Prompt Align Average: 7.96 -> 8.96 | Improved |
| randomly replace some detailed captions with base prompt (vs never or always) (ablation) | MusicInfuser · Prompt alignment | Prompt Align Average: 8.58 -> 8.96 | Improved |
| use higher LoRA rank (ablation) | MusicInfuser · VideoLLaMA 2 judge | Overall: 8.21 -> 8.33 | Improved |
| add LoRA adaptation (ablation) | MusicInfuser · VideoLLaMA 2 judge | Overall: 8.18 -> 8.33 | Improved |
| adapt Mochi with music (MusicInfuser) vs base Mochi | Mochi · Prompt alignment | Style Capture: 7.98 -> 7.80 | Mixed |
Closest papers by meaning
- MoMu-Diffusion: On Learning Long-Term Motion-Music Synchronization and Correspondence (NeurIPS 2024 · similarity 0.89)
- Bidirectional Autoregessive Diffusion Model for Dance Generation (CVPR 2024 · similarity 0.89)
- Live Music Diffusion Models: Efficient Fine-Tuning and Post-Training of Interactive Diffusion Music Generators (NeurIPS 2026 · similarity 0.88)
- V2Meow: Meowing to the Visual Beat via Video-to-Music Generation (AAAI 2024 · similarity 0.88)
- DanceAnyWay: Synthesizing Beat-Guided 3D Dances with Randomized Temporal Contrastive Learning (AAAI 2024 · similarity 0.88)
- MeLFusion: Synthesizing Music from Image and Language Cues using Diffusion Models (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
