AI Paper Atlas

Research map / Vision-language and multimodal / Cross-modal learning

Joint Learning of Audio–Visual Saliency Prediction and Sound Source Localization on Multi-face Videos

Minglang Qiao, Yufan Liu, Mai Xu, Xin Deng, Bing Li, Weiming Hu, Ali Borji

IJCV 2023 · Vision-language and multimodal › Cross-modal learning · approach: Audio-visual source localization · 25 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add optical flow and RGB visual branch over average baseline (ablation)VAM-Net · MVVA saliency ablationCC: 0.364 -> 0.682Improved
add LSTM to visual (RGB+flow) branch (ablation)VAM-Net · MVVA saliency ablationCC: 0.682 -> 0.702Improved
add face branch to visual model (ablation)VAM-Net · MVVA saliency ablationCC: 0.702 -> 0.725Improved
add audio branch to visual+face model (ablation)VAM-Net · MVVA saliency ablationCC: 0.725 -> 0.741Improved
add audio branch to visual+face model (ablation)VAM-Net · MVVA saliency ablationKL: 0.834 -> 0.783Improved
add face branch to visual+LSTM model (ablation)VAM-Net · MVVA saliency ablationAUC: 0.911 -> 0.903Mixed
use full multi-task STMG model instead of conference VASMVAM-Net · MVVA saliencyCC: 0.722 -> 0.741Improved
use full model instead of VASM on Coutrot III KLVAM-Net · Coutrot III saliencyKL: 1.375 -> 1.382No clear effect
use STMG speaking-class output instead of ASCVAM-Net · MVVA sound source localizationIoU: 24.93 -> 52.01Improved
use STMG output instead of ASC for classification accuracyVAM-Net · MVVA sound source localizationAcc: 67.14 -> 78.49Improved
use STMG instead of Owens MSE for AUC-SVAM-Net · MVVA sound source localizationAUC-S: 53.22 -> 42.84Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy