Research map / Video / Video segmentation and tracking
CASP: Consistency-aware Audio-induced Saliency Prediction Model for Omnidirectional Video
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CASP, a query-based audio-induced saliency model for 360-degree videos that fuses spatial audio and ERP visual features with learnable audio queries and a temporal consistency loss.
Problem
Audio-visual saliency prediction for omnidirectional videos is underexplored; existing models treat audio location and attributes separately (via audio energy maps) and ignore temporal consistency of salient regions across frames.
Approach
A query-based audio-visual transformer where anchor audio queries derived from spatially sampled ambisonics are fused with ERP visual features via spatial and temporal audio-visual attention, plus learnable queries, and an adaptive inter-frame cosine-similarity consistency loss.
Key results
- On AVS-ODV (ambisonics), CASP achieves SIM 0.444, CC 0.604, AUC-J 0.949, sAUC 0.813, NSS 4.545, outperforming AVS360 (SIM 0.439, CC 0.595, NSS 3.477).
- On D-SAV360, CASP reaches CC 0.472 and SIM 0.390, versus 0.439 CC and 0.367 SIM for the next-best AVS360.
- Removing the consistency loss lowers mono-audio CC from 0.610 to 0.567 and NSS from 4.585 to 4.251 in the ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CASP instead of the strongest reported baseline | Audio-visual saliency prediction · AVS-ODV ambisonics | SIM: 0.439 -> 0.444 | Helped |
| Use CASP instead of the strongest reported baseline | Audio-visual saliency prediction · AVS-ODV mono | SIM: 0.452 -> 0.461 | Helped |
| Use CASP instead of the strongest reported baseline | Audio-visual saliency prediction · AVS-ODV mute | SIM: 0.456 -> 0.457 | Helped |
| Use CASP instead of the strongest reported baseline | Audio-visual saliency prediction · D-SAV360 | CC: 0.439 -> 0.472 | Helped |
| Replace the pretrained encoder with ResNet18 (ablation) | CASP · AVS-ODV ambisonics | SIM: 0.444 -> 0.413 | Hurt |
| Add the audio-visual interaction module (ablation) | CASP · AVS-ODV ambisonics | SIM: 0.377 -> 0.444 | Helped |
| Use audio queries instead of video queries (ablation) | CASP · AVS-ODV ambisonics | SIM: 0.426 -> 0.444 | Helped |
| Add the consistency loss (ablation) | CASP · AVS-ODV mono | CC: 0.567 -> 0.610 | Helped |
| Set anchor query count to 1 instead of 4 (ablation) | CASP · AVS-ODV ambisonics | SIM: 0.444 -> 0.431 | Hurt |
| Set anchor query count to 2 instead of 4 (ablation) | CASP · AVS-ODV ambisonics | SIM: 0.444 -> 0.427 | Hurt |
| Set anchor query count to 3 instead of 4 (ablation) | CASP · AVS-ODV ambisonics | SIM: 0.444 -> 0.439 | Hurt |
| Set anchor query count to 8 instead of 4 (ablation) | CASP · AVS-ODV ambisonics | SIM: 0.444 -> 0.432 | Hurt |
Surprising findings
- Removing the pretrained encoder slightly increases ambisonics sAUC (0.820 vs 0.813), which the authors attribute to heavier reliance on audio cues.
- The ablation with 1 to 8 anchor queries shows the best SIM and CC at N=4 rather than monotonically improving with more anchors.
Limitations
- Real-time applicability is not demonstrated; the authors list scalability for real-time use as future work.
- Several metrics are not computable on D-SAV360 since fixation maps are unavailable, so only CC and SIM are reported there.
- Only two audio-backbone variants and a fixed number of anchor queries (N=4, Nq=300) are explored.
Experimental setup
Datasets: AVS-ODV D-SAV360
Benchmarks: AVS-ODV D-SAV360
Metrics: SIM CC AUC-J sAUC NSS
Baselines: TASED STSA DAVE SST-Sal AT-Sal AAM SSSL SVGC AVS360 ATSal
Builds on: Find-CMP / EPSNet AVS360 SoundNet ResNet-18
Closest papers by meaning
- DiffSal: Joint Audio and Video Learning for Diffusion Saliency Prediction (CVPR 2024 · similarity 0.88)
- From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation (ICML 2024 · similarity 0.88)
- Learning to Highlight Audio by Watching Movies (CVPR 2025 · similarity 0.87)
- Adapting to the Unknown: Training-Free Audio-Visual Event Perception with Dynamic Thresholds (CVPR 2025 · similarity 0.87)
- Robust Audio-Visual Segmentation via Audio-Guided Visual Convergent Alignment (CVPR 2025 · similarity 0.86)
- Unveiling the Power of Audio-Visual Early Fusion Transformers with Dense Interactions through Masked Modeling (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.