Research map / Video / Video generation
Hear What You See: Video-to-Audio Generation with Diffusion Transformer and Semantic-Temporal Alignment-Ranked Direct Preference Optimization
Kai Wang, Tao Zhou, Jiayi Lei, Jing Wang, Jinman Zhao, Weiguo Pian, Yuan Cheng, Yapeng Tian, Peng Gao, Bin Fu, Yihao Liu, Dimitrios Hatzinakos and 1 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply STAR-DPO preference optimization to VisioSonic Base (ablation) | VisioSonic Base · VGGSound test | FD PaSST: 58.27 -> 55.48 | Improved |
| apply STAR-DPO preference optimization to VisioSonic Base (ablation) | VisioSonic Base · VGGSound test | DeSync: 0.45 -> 0.41 | Improved |
| use co-attention with concatenated audio-video latents instead of MMDiT blocks (ablation) | VisioSonic Base · VGGSound test | FD PaSST: 60.14 -> 58.27 | Improved |
| use Flux-like blocks instead of shared co-attention (ablation) | VisioSonic Base · VGGSound test | FD PaSST: 58.27 -> 66.76 | Hurt |
| use descriptive captions instead of class labels as text (ablation) | VisioSonic Base · VGGSound test | FD PaSST: 62.72 -> 58.27 | Improved |
| fuse audio and video first (AV) then text, rather than separate streams (ablation) | VisioSonic Base · VGGSound test | IB-score: 31.0 -> 32.8 | Improved |
| fuse audio and video first (AV) rather than text-video then audio (ablation) | VisioSonic Base · VGGSound test | DeSync: 0.48 -> 0.45 | Improved |
| use IB-AV + DeSync joint reward instead of CAVP + AV-Align (ablation) | STAR-DPO · VGGSound test | FD PaSST: 58.34 -> 55.48 | Improved |
| use DeSync (Synchformer) instead of AV-Align as temporal reward (ablation) | STAR-DPO · VGGSound test | DeSync: 0.44 -> 0.41 | Improved |
| run a third DPO iteration after the second (ablation) | STAR-DPO · VGGSound test | FD PaSST: 55.48 -> 56.65 | Hurt |
| rank jointly on semantic and temporal reward instead of cascading DeSync then IB-AV (ablation) | STAR-DPO · VGGSound test | FD PaSST: 68.57 -> 56.09 | Improved |
| cascaded DeSync-first ranking (ablation) | STAR-DPO · VGGSound test | DeSync: 0.44 -> 0.38 | Mixed |
Closest papers by meaning
- Native Audio-Visual Alignment for Generation (NeurIPS 2026 · similarity 0.91)
- Foley-Flow: Coordinated Video-to-Audio Generation with Masked Audio-Visual Alignment and Dynamic Conditional Flows (CVPR 2025 · similarity 0.91)
- Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Aligners (CVPR 2024 · similarity 0.90)
- Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation (AAAI 2024 · similarity 0.90)
- Model-Guided Dual-Role Alignment for High-Fidelity Open-Domain Video-to-Audio Generation (NeurIPS 2025 · similarity 0.90)
- MMDisCo: Multi-Modal Discriminator-Guided Cooperative Diffusion for Joint Audio and Video Generation (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
