AI Paper Atlas

Research map / Vision-language and multimodal / Cross-modal learning

Visually Informed Binaural Audio Generation without Binaural Audios

Xudong Xu, Hang Zhou, Ziwei Liu, Bo Dai, Xiaogang Wang, Dahua Lin

CVPR 2021 · Vision-language and multimodal › Cross-modal learning · approach: Audio-visual source localization · 57 citations (OpenAlex)

Check your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
train from pseudo visual-stereo pairs using mono data, without binaural recordingsPseudoBinaural · FAIR-Play and MUSIC-StereoSTFT: 1.024 -> 0.944; 1.014 -> 0.943Improved
add separation training (ablation)PseudoBinaural · FAIR-Play and MUSIC-StereoSTFT: 0.951 -> 0.944; 0.953 -> 0.943Improved
augment supervised training with pseudo visual-stereo pairsbinaural audio training · FAIR-Play and MUSIC-StereoSTFT: 0.917 -> 0.878; 0.942 -> 0.891Improved
train on mono-derived pseudo pairs instead of recorded binaural pairsPseudoBinaural vs. Mono2Binaural · FAIR-Play, MUSIC-Stereo, and YT-MUSIC cross-dataset evaluationSTFT: 0.996 -> 0.959; 0.971 -> 0.952; 0.717 -> 0.653Improved
combine ambisonic decomposition with HRIR decoding rather than either decoding scheme alone (ablation)PseudoBinaural · FAIR-Play and MUSIC-StereoSTFTImproved
vary the number of mixed mono videos instead of using a fixed K (ablation)PseudoBinaural · FAIR-PlaySTFT: 0.965; 0.935; 0.967 -> 0.878Improved
set the border azimuth angle to π/3 rather than the tested alternatives (ablation)PseudoBinaural visual-coordinate mapping · FAIR-PlaySTFT: 0.923; 0.896; 0.878; 0.884; 0.886Improved
use the empirical mix ratio 4:5:1 rather than other tested ratios (ablation)PseudoBinaural · FAIR-PlaySTFT: 0.880; 0.878; 0.884; 0.885No clear effect
use only 1/3 of the ground-truth binaural audios with pseudo-pair augmentation (ablation)Augment-PseudoBinaural · FAIR-Playrelative performance: 1/3No clear effect
vary the separation-loss weight within [0.5, 1.5] (ablation)PseudoBinaural · not specifiedoverall performance: [0.5, 1.5]No clear effect
augment supervised training with pseudo pairsbinaural audio training · Youtube-ASMRSTFT: 0.198 -> 0.196Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy