Research map / Video / Video generation
DiffSal: Joint Audio and Video Learning for Diffusion Saliency Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiffSal, a conditional diffusion model that predicts audio-visual saliency maps by denoising saliency maps conditioned on video and audio features via a Saliency-UNet with multi-modal attention modulation, trained with a simple MSE loss.
Problem
Audio-visual saliency prediction is limited by customized architectures and task-specific losses, and it is unclear how to design a diffusion model that captures effective audio-visual interaction and a generalized saliency network.
Approach
Formulates saliency prediction as conditional generation of the saliency map from noise, conditioned on spatio-temporal video and audio features, with a Saliency-UNet that uses efficient spatio-temporal cross-attention and a multi-modal interaction module, optimized with plain MSE and iteratively refined at inference.
Key results
- DiffSal reaches CC 0.738 and SIM 0.571 on AVAD, versus 0.691 and 0.528 for CASP-Net.
- DiffSal reports an average relative improvement of 6.3% over the previous state-of-the-art across six benchmarks.
- Replacing the cross-entropy or KL losses with plain MSE improves AVAD CC from 0.690 (CE) and 0.720 (KL) to 0.738.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DiffSal instead of CASP-Net | AVSP models · DIEM | CC: 0.655 -> 0.660 | Helped |
| Use DiffSal instead of CASP-Net | AVSP models · Coutrot1 | CC: 0.561 -> 0.638 | Helped |
| Use DiffSal instead of CASP-Net | AVSP models · Coutrot2 | CC: 0.788 -> 0.835 | Helped |
| Use DiffSal instead of CASP-Net | AVSP models · AVAD | CC: 0.691 -> 0.738 | Helped |
| Use DiffSal instead of CASP-Net | AVSP models · ETMD | CC: 0.620 -> 0.652 | Helped |
| Use DiffSal instead of CASP-Net | AVSP models · SumMe | CC: 0.499 -> 0.572 | Helped |
| Add ECA and MIM to the baseline (ablation) | DiffSal · AVAD | CC: 0.701 -> 0.738 | Helped |
| Add ECA and MIM to the baseline (ablation) | DiffSal · ETMD | CC: 0.632 -> 0.652 | Helped |
| Use audio-only inputs instead of audio-visual inputs (ablation) | DiffSal · AVAD | CC: 0.738 -> 0.343 | Hurt |
| Use video-only inputs instead of audio-visual inputs (ablation) | DiffSal · AVAD | CC: 0.738 -> 0.716 | Hurt |
| Replace MIM with bilinear interaction (ablation) | DiffSal · AVAD | CC: 0.738 -> 0.716 | Hurt |
| Replace MIM with addition (ablation) | DiffSal · AVAD | CC: 0.738 -> 0.706 | Hurt |
Surprising findings
- A simple MSE objective outperforms task-tailored CE and KL losses for audio-visual saliency prediction.
- The audio-only variant of DiffSal performs far worse than the video-only variant (AVAD CC 0.343 vs 0.716).
Limitations
- Iterative denoising requires multiple network passes at inference (4 steps used), increasing cost relative to a single forward pass.
- Parameter count rises to 76.57M, above several baselines, and FLOPs are second-highest among compared methods.
Experimental setup
Datasets: AVAD Coutrot1 Coutrot2 DIEM ETMD SumMe DHF1k Kinetics AudioSet
Benchmarks: AVAD Coutrot1 Coutrot2 DIEM ETMD SumMe
Metrics: CC NSS AUC-J SIM
Baselines: ACLNet TASED-Net STAViS ViNet TSFP-Net CASP-Net
Builds on: DDPM DDIM DiffusionDet MViTv2 VGGish
Closest papers by meaning
- CASP: Consistency-aware Audio-induced Saliency Prediction Model for Omnidirectional Video (CVPR 2025 · similarity 0.88)
- A Versatile Diffusion Transformer with Mixture of Noise Levels for Audiovisual Generation (NeurIPS 2024 · similarity 0.87)
- CaRDiff: Video Salient Object Ranking Chain of Thought Reasoning for Saliency Prediction with Diffusion (AAAI 2025 · similarity 0.87)
- Diff-2-in-1: Bridging Generation and Dense Perception with Diffusion Models (ICLR 2025 · similarity 0.87)
- MMDisCo: Multi-Modal Discriminator-Guided Cooperative Diffusion for Joint Audio and Video Generation (ICLR 2025 · similarity 0.85)
- Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Aligners (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.