Research map / Video / Video generation
Unison: Harmonizing Motion, Speech, and Sound for Human-Centric Audio-Video Generation
Shihao Cheng, Jiaxu Zhang, Quanyue Song, Shansong Liu, Guo Zhi, Xiao-Lei Zhang, Chi Zhang, Xuelong Li, Zhigang Tu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove semantic-guided harmonization (decoupled speech/SFX streams) (ablation) | Unison · TI2AV test set | PQ: 6.12 -> 6.34 | Improved |
| add Bi-ACA between speech and SFX streams (ablation) | Unison · TI2AV test set | PQ: 6.20 -> 6.34 | Improved |
| add semantic-conditioned gating (ablation) | Unison · TI2AV test set | PQ: 6.21 -> 6.34 | Improved |
| add cross-modal forcing strategy (ablation) | Unison · TI2AV test set | DS: 0.19 -> 0.08 | Improved |
| add cross-modal forcing (ablation) | Unison · TI2AV test set | VA: 3.91 -> 4.02 | Improved |
| audio-side modules (SGHS, Bi-ACA, SCG) on video quality (ablation) | Unison · TI2AV test set | VA: 3.99-4.01 -> 4.02 | No clear effect |
| progressive three-phase schedule instead of synchronous timesteps (ablation) | Unison · TI2AV test set | DS: 0.17 -> 0.08 | Improved |
| progressive schedule instead of independent timesteps without warmup (ablation) | Unison · TI2AV test set | LSE-C: 3.28 -> 3.30 | Improved |
| Unison vs LTX-2 on visual aesthetics | Unison · TI2AV test set | VA: 4.15 -> 4.02 | Hurt |
| Unison vs LTX-2 in user study lip-sync | Unison · user study | lip-sync mean rank: 1.74 -> 1.86 | Hurt |
Closest papers by meaning
- Native Audio-Visual Alignment for Generation (NeurIPS 2026 · similarity 0.91)
- Audio-Sync Video Generation with Multi-Stream Temporal Control (NeurIPS 2025 · similarity 0.89)
- VinTAGe: Joint Video and Text Conditioning for Holistic Audio Generation (CVPR 2025 · similarity 0.88)
- MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis (CVPR 2025 · similarity 0.88)
- UniFlow-Audio: Unified Flow Matching for Audio Generation from Omni-Modalities (NeurIPS 2026 · similarity 0.88)
- DiVAS: Video and Audio Synchronization with Dynamic Frame Rates (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
