Research map / Vision-language and multimodal / Cross-modal learning
Learning to Predict Salient Faces: A Novel Visual-Audio Saliency Model
Yufan Liu, Minglang Qiao, Mai Xu, Bing Li, Weiming Hu, Ali Borji
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add convolutional LSTM to the RGB+flow visual branch (ablation) | visual branch · MVVA | CC: 0.632 -> 0.671 | Improved |
| add optical flow to the RGB visual branch (ablation) | visual branch · MVVA | CC: 0.527 -> 0.632 | Improved |
| add audio branch to the visual branch (ablation) | visual branch · MVVA | KL: 0.971 -> 0.843 | Improved |
| add audio branch to the face branch (ablation) | face branch · MVVA | CC: 0.569 -> 0.609 | Improved |
| add face branch to the visual-audio model (ablation) | visual+audio · MVVA | CC: 0.712 -> 0.722 | Improved |
| add face branch to the visual-audio model (ablation) | visual+audio · MVVA | KL: 0.843 -> 0.823 | Improved |
| add visual branch to face+audio (ablation) | face+audio · MVVA | CC: 0.609 -> 0.722 | Improved |
| use the full multi-modal model instead of best competitor on MVVA | Visual+audio+face · MVVA | NSS: 3.495 -> 3.976 | Improved |
| evaluate full model on unseen Coutrot II | Visual+audio+face · Coutrot II | KL: 1.031 -> 0.915 | Improved |
| use face branch alone (ablation) | face branch · MVVA | CC: 0.722 -> 0.569 | Hurt |
Closest papers by meaning
- Joint Learning of Audio–Visual Saliency Prediction and Sound Source Localization on Multi-face Videos (IJCV 2023 · similarity 0.93)
- Predicting Salient Face in Multiple-Face Videos (CVPR 2017 · similarity 0.91)
- CASP: Consistency-aware Audio-induced Saliency Prediction Model for Omnidirectional Video (CVPR 2025 · similarity 0.90)
- DiffSal: Joint Audio and Video Learning for Diffusion Saliency Prediction (CVPR 2024 · similarity 0.89)
- CaRDiff: Video Salient Object Ranking Chain of Thought Reasoning for Saliency Prediction with Diffusion (AAAI 2025 · similarity 0.87)
- From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
