Research map / Video / Video segmentation and tracking
Spherical Vision Transformers for Audio-Visual Saliency Prediction in 360$^{\circ }$∘ Videos
Mert Cokelek, Halit Ozsoy, Nevrez İmamoğlu, Cagri Ozcinar, İnci Ayhan, Erkut Erdem, Aykut Erdem
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add spherical position embeddings in place of 1D learnable embeddings (ablation) | SalViT360 (VSA) · VR-EyeTracking | NSS: 2.518 -> 2.575 | Improved |
| add viewport spatio-temporal attention (VSTA) over spatial-only VSA (ablation) | SalViT360 · VR-EyeTracking | NSS: 2.575 -> 2.664 | Improved |
| add viewport spatial attention over no attention (ablation) | SalViT360 · VR-EyeTracking | NSS: 2.306 -> 2.575 | Improved |
| add VAC loss with mask on top of VSTA (ablation) | SalViT360 · VR-EyeTracking | KLD: 6.174 -> 5.744 | Improved |
| add mask to the VAC loss (ablation) | SalViT360 · VR-EyeTracking | KLD: 6.011 -> 5.744 | Improved |
| add VAC loss without mask on NSS (ablation) | SalViT360 · VR-EyeTracking | NSS: 2.664 -> 2.624 | Hurt |
| use VSTA instead of 2+1D CNN encoder (ablation) | SalViT360 · VR-EyeTracking | NSS: 2.568 -> 2.664 | Improved |
| use VSTA instead of offline EMA aggregation (ablation) | SalViT360 · VR-EyeTracking | NSS: 2.591 -> 2.664 | Improved |
| use VSTA instead of 2+1D CNN on KLD (ablation) | SalViT360 · VR-EyeTracking | KLD: 5.915 -> 6.174 | Hurt |
| use ambisonics instead of mute ground-truth saliency | SalViT360 · YT360-EyeTracking | NSS: 2.115 -> 2.346 | Improved |
| add spatial audio branch to visual model | SalViT360 -> SalViT360-AV · YT360-EyeTracking | NSS: 2.346 -> 2.449 | Improved |
| use ambisonic encoding instead of raw FOA input (ablation) | SalViT360-AV · YT360-EyeTracking | NSS: 2.443 -> 2.449 | No clear effect |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- DiffSal — DiffSal: Joint Audio and Video Learning for Diffusion Saliency Prediction (CVPR 2024) · better on 4 of 4 shared results
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- CASP: Consistency-aware Audio-induced Saliency Prediction Model for Omnidirectional Video (CVPR 2025 · similarity 0.91)
- AV-Cloud: Spatial Audio Rendering Through Audio-Visual Cloud Splatting (NeurIPS 2024 · similarity 0.85)
- Textured Mesh Saliency: Bridging Geometry and Texture for Human Perception in 3D Graphics (AAAI 2025 · similarity 0.85)
- ImViD: Immersive Volumetric Videos for Enhanced VR Engagement (CVPR 2025 · similarity 0.84)
- OmniAudio: Generating Spatial Audio from 360-Degree Video (ICML 2025 · similarity 0.84)
- DiffSal: Joint Audio and Video Learning for Diffusion Saliency Prediction (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
