Research map / Vision-language and multimodal / Cross-modal learning
Audio-Visual Semantic Graph Network for Audio-Visual Event Localization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AVSGN, which uses shared text labels from frozen CLIP/CLAP encoders with attention-based cross-modal semantic alignment and a three-subgraph cross-modal graph interaction module for audio-visual event localization on AVE.
Problem
Audio-visual event localization (AVEL) suffers from semantic inconsistency between heterogeneous audio and visual modalities and from weak cross-temporal dependency modeling.
Approach
Introduces shared semantic text labels (from CLIP and CLAP text encoders) as a bridge, aligns visual, audio and text features via attention in a shared space (CMSA), and disentangles cross-modal interactions into audio-text, audio-visual and visual-text GCN subgraphs (CMGI).
Key results
- Achieves 83.1% fully-supervised accuracy on AVE, versus 82.2% for DG-SCT and 78.6% for LAVISH.
- Achieves 78.8% weakly-supervised accuracy on AVE, versus 76.8% for CSS-Net.
- Ablation: removing the A↔V subgraph drops fully-supervised accuracy from 83.1% to 80.3%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add AVSGN with CMSA and CMGI instead of DG-SCT | AVE localization with Swin-V2-L and HTS-AT · AVE, fully-supervised | accuracy: 82.2 -> 83.1 | Helped |
| Add AVSGN instead of CSS-Net | AVE localization · AVE, weakly-supervised | accuracy: 76.8 -> 78.8 | Helped |
| Add pseudo intra-modal alignment (ablation) | AVSGN · AVE, fully-supervised and weakly-supervised | accuracy: 82.8 -> 83.1; 78.5 -> 78.8 | Helped |
| Add cross-modal visual alignment (ablation) | AVSGN · AVE, fully-supervised and weakly-supervised | accuracy: 80.4 -> 83.1; 76.0 -> 78.8 | Helped |
| Add cross-modal audio alignment (ablation) | AVSGN · AVE, fully-supervised and weakly-supervised | accuracy: 81.8 -> 83.1; 77.2 -> 78.8 | Helped |
| Add cross-modal text alignment (ablation) | AVSGN · AVE, fully-supervised and weakly-supervised | accuracy: 82.3 -> 83.1; 77.8 -> 78.8 | Helped |
| Add audio-visual subgraph (ablation) | AVSGN CMGI · AVE, fully-supervised and weakly-supervised | accuracy: 80.3 -> 83.1; 75.9 -> 78.8 | Helped |
| Add visual-text subgraph (ablation) | AVSGN CMGI · AVE, fully-supervised and weakly-supervised | accuracy: 81.7 -> 83.1; 76.6 -> 78.8 | Helped |
| Add audio-text subgraph (ablation) | AVSGN CMGI · AVE, fully-supervised and weakly-supervised | accuracy: 82.5 -> 83.1; 78.0 -> 78.8 | Helped |
| Add audio-visual gate loss (ablation) | AVSGN · AVE, fully-supervised | accuracy: 81.7 -> 83.1 | Helped |
| Add semantic consistency loss (ablation) | AVSGN · AVE, fully-supervised | accuracy: 82.0 -> 83.1 | Helped |
Limitations
- Evaluated only on the AVE dataset
- Relies on large frozen pretrained encoders (Swin-V2-L CLIP, HTS-AT CLAP) for its gains over VGG-based baselines
Experimental setup
Datasets: AVE
Benchmarks: AVE
Metrics: segment-level classification accuracy
Baselines: AVEL CMBS CSS-Net LAVISH DG-SCT VSCG DMIN CMRAN
Builds on: CLIP CLAP AVEL CMBS LAVISH
Closest papers by meaning
- Towards Open-Vocabulary Audio-Visual Event Localization (CVPR 2025 · similarity 0.92)
- Dense Audio-Visual Event Localization Under Cross-Modal Consistency and Multi-Temporal Granularity Collaboration (AAAI 2025 · similarity 0.91)
- Multimodal Class-aware Semantic Enhancement Network for Audio-Visual Video Parsing (AAAI 2025 · similarity 0.89)
- LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videos (CVPR 2025 · similarity 0.89)
- Empowering LLMs with Pseudo-Untrimmed Videos for Audio-Visual Temporal Understanding (AAAI 2025 · similarity 0.88)
- PreFM: Online Audio-Visual Event Parsing via Predictive Future Modeling (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.