Research map / Vision-language and multimodal / Cross-modal learning
Multimodal Class-aware Semantic Enhancement Network for Audio-Visual Video Parsing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MM-CSE, which decouples holistic audio/visual features into class-wise (event + background) features and enhances them via co-occurrence and local-global fusion for audio-visual video parsing on LLP.
Problem
Audio-visual video parsing suffers from semantic interference when intra- and cross-modal temporal interactions use semantically mixed holistic segment features, since each segment may contain multiple events.
Approach
Decouple each segment's holistic audio/visual feature into K event-specific features plus a background feature via independent linear layers, then model event co-occurrence within segments (SECM) and fuse global video semantics (LGSF) so each class aggregates only matched semantics from other segments.
Key results
- With VGGish/ResNet-152/R(2+1)D features, MM-CSE reaches 61.5% average over ten metrics on LLP, versus 60.5% for VALOR.
- With CLAP and CLIP features, MM-CSE* reaches 65.4% average, versus 63.2% for VALOR (+2.2%).
- Removing the SECM block drops the average from 65.4 to 61.6 in the CLAP/CLIP setting; removing cross-FGSE drops it to 60.6.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace VALOR with MM-CSE using VGGish, ResNet-152, and R(2+1)D features | VALOR on LLP · LLP test | Avg. F-score: 60.5 -> 61.5 | Helped |
| replace VALOR* with MM-CSE* using CLAP and CLIP features | VALOR* on LLP · LLP test | Avg. F-score: 63.2 -> 65.4 (+2.2) | Helped |
| add event-specific class feature decoupling (ablation) | MM-CSE · LLP test | Avg. F-score: 63.0 -> 64.2 | Helped |
| add a background class to event-specific feature decoupling (ablation) | MM-CSE · LLP test | Avg. F-score: 64.2 -> 65.4 | Helped |
| add SECM (ablation) | MM-CSE FGSE module · LLP test | Avg. F-score: 63.2 -> 65.4 | Helped |
| add LGSF (ablation) | MM-CSE FGSE module · LLP test | Avg. F-score: 61.6 -> 65.4 | Helped |
| add intra-FGSE (ablation) | MM-CSE FGSE module · LLP test | Avg. F-score: 64.1 -> 65.4 | Helped |
| add cross-FGSE (ablation) | MM-CSE FGSE module · LLP test | Avg. F-score: 60.6 -> 65.4 | Helped |
| add reconstruction loss alone (ablation) | MM-CSE loss objective · LLP test | Avg. F-score: 64.3 -> 64.3 | No clear effect |
| add orthogonality loss alone (ablation) | MM-CSE loss objective · LLP test | Avg. F-score: 64.3 -> 64.4 | No clear effect |
| combine reconstruction and orthogonality losses (ablation) | MM-CSE loss objective · LLP test | Avg. F-score: 64.3 -> 64.8 | Helped |
| add event co-occurrence loss to the combined reconstruction and orthogonality losses (ablation) | MM-CSE loss objective · LLP test | Avg. F-score: 64.8 -> 65.4 | Helped |
Surprising findings
- Decoupling into event classes alone without the background class yields 64.2 vs 65.4 with background, indicating background modeling matters.
- Using L_rec or L_ort alone gives no benefit over basic loss; only their combination helps.
Limitations
- Higher FLOPs and runtime than VALOR due to fine-grained class-wise processing.
- Evaluated only on the LLP dataset, described as the sole standard AVVP dataset.
Experimental setup
Datasets: LLP
Benchmarks: LLP
Metrics: F-score segment-level F-score event-level Type@AV Event@AV
Baselines: HAN CVCMS MA MGN MM-Pyr JoMoLD DHHN CMPAE LSLD VALOR LFAV LEAP
Builds on: HAN VALOR MMIL pooling CLIP CLAP
Closest papers by meaning
- PreFM: Online Audio-Visual Event Parsing via Predictive Future Modeling (NeurIPS 2025 · similarity 0.90)
- Audio-Visual Semantic Graph Network for Audio-Visual Event Localization (CVPR 2025 · similarity 0.89)
- Towards Open-Vocabulary Audio-Visual Event Localization (CVPR 2025 · similarity 0.88)
- UWAV: Uncertainty-weighted Weakly-supervised Audio-Visual Video Parsing (CVPR 2025 · similarity 0.88)
- LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videos (CVPR 2025 · similarity 0.88)
- Dense Audio-Visual Event Localization Under Cross-Modal Consistency and Multi-Temporal Granularity Collaboration (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.