Research map / Vision-language and multimodal / Cross-modal learning
Joint Learning of Audio–Visual Saliency Prediction and Sound Source Localization on Multi-face Videos
Minglang Qiao, Yufan Liu, Mai Xu, Xin Deng, Bing Li, Weiming Hu, Ali Borji
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add optical flow and RGB visual branch over average baseline (ablation) | VAM-Net · MVVA saliency ablation | CC: 0.364 -> 0.682 | Improved |
| add LSTM to visual (RGB+flow) branch (ablation) | VAM-Net · MVVA saliency ablation | CC: 0.682 -> 0.702 | Improved |
| add face branch to visual model (ablation) | VAM-Net · MVVA saliency ablation | CC: 0.702 -> 0.725 | Improved |
| add audio branch to visual+face model (ablation) | VAM-Net · MVVA saliency ablation | CC: 0.725 -> 0.741 | Improved |
| add audio branch to visual+face model (ablation) | VAM-Net · MVVA saliency ablation | KL: 0.834 -> 0.783 | Improved |
| add face branch to visual+LSTM model (ablation) | VAM-Net · MVVA saliency ablation | AUC: 0.911 -> 0.903 | Mixed |
| use full multi-task STMG model instead of conference VASM | VAM-Net · MVVA saliency | CC: 0.722 -> 0.741 | Improved |
| use full model instead of VASM on Coutrot III KL | VAM-Net · Coutrot III saliency | KL: 1.375 -> 1.382 | No clear effect |
| use STMG speaking-class output instead of ASC | VAM-Net · MVVA sound source localization | IoU: 24.93 -> 52.01 | Improved |
| use STMG output instead of ASC for classification accuracy | VAM-Net · MVVA sound source localization | Acc: 67.14 -> 78.49 | Improved |
| use STMG instead of Owens MSE for AUC-S | VAM-Net · MVVA sound source localization | AUC-S: 53.22 -> 42.84 | Hurt |
Closest papers by meaning
- Learning to Predict Salient Faces: A Novel Visual-Audio Saliency Model (ECCV 2020 · similarity 0.93)
- CASP: Consistency-aware Audio-induced Saliency Prediction Model for Omnidirectional Video (CVPR 2025 · similarity 0.89)
- DiffSal: Joint Audio and Video Learning for Diffusion Saliency Prediction (CVPR 2024 · similarity 0.87)
- From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation (ICML 2024 · similarity 0.87)
- Unveiling the Power of Audio-Visual Early Fusion Transformers with Dense Interactions through Masked Modeling (CVPR 2024 · similarity 0.86)
- Prompting Segmentation with Sound Is Generalizable Audio-Visual Source Localizer (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
