Research map / Video / Video-language understanding
Dense Audio-Visual Event Localization Under Cross-Modal Consistency and Multi-Temporal Granularity Collaboration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CCNet for dense audio-visual event localization in untrimmed videos, combining a cross-modal consistency collaboration module (CMCC) and a multi-temporal granularity collaboration module (MTGC) to reach a new state of the art on UnAV-100.
Problem
Dense audio-visual event localization (DAVEL) in long untrimmed videos requires temporally localizing all events that co-occur in audio and visual streams, with overlapping classes and varied durations, which short-video AVEL methods cannot handle.
Approach
A cross-modal interaction branch plus a temporal consistency-gated branch enforce audio-visual event consistency, and bidirectional coarse-to-fine and fine-to-coarse attention across a feature pyramid of temporal granularities support localization of events with varied durations.
Key results
- On UnAV-100, CCNet reaches 49.2% average mAP versus 47.8% for the DAVEL baseline (51.0% for DAVEL vs 54.1% for CCNet with ONE-PEACE features).
- On ActivityNet1.3, CCNet achieves 38.6% average mAP, above UniAV's 36.2%.
- On LFAV, CCNet improves average F1 from 24.52 to 26.34.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CCNet instead of DAVEL | DAVEL on UnAV-100 · UnAV-100 | mAP (Avg.): 47.8 -> 49.2 | Helped |
| Use CCNet instead of UniAV with ONE-PEACE features | UniAV on UnAV-100 · UnAV-100 (ONE-PEACE features) | mAP (Avg.): 51.7 -> 54.1 | Helped |
| Use CCNet instead of UniAV | UniAV on ActivityNet1.3 · ActivityNet1.3 | mAP (Avg.): 36.2 -> 38.6 | Helped |
| Use CCNet instead of DAVEL | DAVEL on LFAV · LFAV | event-level F1 (Avg.): 24.52 -> 26.34 | Helped |
| Add both C2F and F2C blocks (ablation) | CCNet with CMCC only on UnAV-100 · UnAV-100 | mAP (Avg.): 47.9 -> 49.2 | Helped |
| Add F2C (ablation) | CCNet with CMCC and C2F on UnAV-100 · UnAV-100 | mAP (Avg.): 48.3 -> 49.2 | Helped |
| Add C2F (ablation) | CCNet with CMCC and F2C on UnAV-100 · UnAV-100 | mAP (Avg.): 48.2 -> 49.2 | Helped |
| Add TCG (ablation) | CCNet with CMI on UnAV-100 · UnAV-100 | mAP (Avg.): 48.1 -> 49.2 | Helped |
| Add CMI (ablation) | CCNet with TCG on UnAV-100 · UnAV-100 | mAP (Avg.): 47.6 -> 49.2 | Helped |
| Apply C2F before F2C instead of F2C before C2F (ablation) | CCNet MTGC on UnAV-100 · UnAV-100 | mAP (Avg.): 47.1 -> 49.2 | Helped |
| Apply CMCC to both audio and visual modalities instead of only one (ablation) | CCNet on UnAV-100 · UnAV-100 | mAP (Avg.): 47.9 -> 49.2 | Helped |
| Set CMCC granularity count to 6 instead of 4, 5, or 7 (ablation) | CCNet on UnAV-100 · UnAV-100 | mAP (Avg.): 47.3; 48.7; 49.2; 48.4 | Mixed |
Surprising findings
- Temporal action localization models (VSGN, TadTR, ActionFormer) lag far behind the DAVEL baseline when adapted to audio-visual input, underscoring the need for cross-modal design.
- Aggregating features from all granularities instead of adjacent ones does not help and slightly lowers average mAP (48.9 vs 49.2).
Limitations
- Gains over the DAVEL baseline are modest (about 1.4 points average mAP on UnAV-100 with standard features).
- Ablations on the MTGC order show that C2F-then-F2C is required; performance is sensitive to design choices such as L_c.
- On ActivityNet1.3 the model is weaker than UniAV at the strict 0.95 tIoU threshold (6.2 vs 6.7).
Experimental setup
Datasets: UnAV-100 ActivityNet1.3 LFAV
Benchmarks: UnAV-100 ActivityNet1.3 LFAV
Metrics: mAP at tIoU 0.5:0.1:0.9 average mAP event-level F1
Baselines: DAVEL VSGN TadTR ActionFormer UniAV AVE CMBS PSP MM-Pyramid Tridet
Builds on: DAVEL multi-head attention focal loss generalized IoU loss Soft-NMS
Closest papers by meaning
- Towards Open-Vocabulary Audio-Visual Event Localization (CVPR 2025 · similarity 0.91)
- Empowering LLMs with Pseudo-Untrimmed Videos for Audio-Visual Temporal Understanding (AAAI 2025 · similarity 0.91)
- Audio-Visual Semantic Graph Network for Audio-Visual Event Localization (CVPR 2025 · similarity 0.91)
- Adapting to the Unknown: Training-Free Audio-Visual Event Perception with Dynamic Thresholds (CVPR 2025 · similarity 0.90)
- LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videos (CVPR 2025 · similarity 0.89)
- Robust Audio-Visual Segmentation via Audio-Guided Visual Convergent Alignment (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.