Research map / Vision-language and multimodal / Cross-modal learning
Visually Informed Binaural Audio Generation without Binaural Audios
Xudong Xu, Hang Zhou, Ziwei Liu, Bo Dai, Xiaogang Wang, Dahua Lin
Check your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train from pseudo visual-stereo pairs using mono data, without binaural recordings | PseudoBinaural · FAIR-Play and MUSIC-Stereo | STFT: 1.024 -> 0.944; 1.014 -> 0.943 | Improved |
| add separation training (ablation) | PseudoBinaural · FAIR-Play and MUSIC-Stereo | STFT: 0.951 -> 0.944; 0.953 -> 0.943 | Improved |
| augment supervised training with pseudo visual-stereo pairs | binaural audio training · FAIR-Play and MUSIC-Stereo | STFT: 0.917 -> 0.878; 0.942 -> 0.891 | Improved |
| train on mono-derived pseudo pairs instead of recorded binaural pairs | PseudoBinaural vs. Mono2Binaural · FAIR-Play, MUSIC-Stereo, and YT-MUSIC cross-dataset evaluation | STFT: 0.996 -> 0.959; 0.971 -> 0.952; 0.717 -> 0.653 | Improved |
| combine ambisonic decomposition with HRIR decoding rather than either decoding scheme alone (ablation) | PseudoBinaural · FAIR-Play and MUSIC-Stereo | STFT | Improved |
| vary the number of mixed mono videos instead of using a fixed K (ablation) | PseudoBinaural · FAIR-Play | STFT: 0.965; 0.935; 0.967 -> 0.878 | Improved |
| set the border azimuth angle to π/3 rather than the tested alternatives (ablation) | PseudoBinaural visual-coordinate mapping · FAIR-Play | STFT: 0.923; 0.896; 0.878; 0.884; 0.886 | Improved |
| use the empirical mix ratio 4:5:1 rather than other tested ratios (ablation) | PseudoBinaural · FAIR-Play | STFT: 0.880; 0.878; 0.884; 0.885 | No clear effect |
| use only 1/3 of the ground-truth binaural audios with pseudo-pair augmentation (ablation) | Augment-PseudoBinaural · FAIR-Play | relative performance: 1/3 | No clear effect |
| vary the separation-loss weight within [0.5, 1.5] (ablation) | PseudoBinaural · not specified | overall performance: [0.5, 1.5] | No clear effect |
| augment supervised training with pseudo pairs | binaural audio training · Youtube-ASMR | STFT: 0.198 -> 0.196 | Improved |
Closest papers by meaning
- Cyclic Learning for Binaural Audio Generation and Localization (CVPR 2024 · similarity 0.89)
- BinauralFlow: A Causal and Streamable Approach for High-Quality Binaural Speech Synthesis with Flow Matching Models (ICML 2025 · similarity 0.87)
- Both Ears Wide Open: Towards Language-Driven Spatial Audio Generation (ICLR 2025 · similarity 0.85)
- OmniAudio: Generating Spatial Audio from 360-Degree Video (ICML 2025 · similarity 0.85)
- Supervising Sound Localization by In-the-wild Egomotion (CVPR 2025 · similarity 0.85)
- Hearing Anything Anywhere (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
