Research map / Vision-language and multimodal / Cross-modal learning
Towards Open-Vocabulary Audio-Visual Event Localization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces open-vocabulary audio-visual event localization (OV-AVEL) with the OV-AVEBench dataset (24,800 videos, 67 classes, 46 seen/21 unseen) and two ImageBind-based baselines (training-free and fine-tuned with temporal Transformer layers).
Problem
Audio-visual event localization is mostly closed-set, and prior open-set work only labels unseen events as "unknown" without category semantics; there is no benchmark for predicting explicit categories for unseen events at segment level.
Approach
Reformulate AVEL as open-vocabulary: use ImageBind's joint embedding space to match segment-level audio and visual features against text class embeddings (seen and unseen), with a lightweight temporal Transformer fine-tuning on seen-class training data to improve localization.
Key results
- Fine-tuning baseline reaches 57.8 average on total test set versus 46.6 for the training-free ImageBind baseline (Table 2).
- On unseen test data the fine-tuned model achieves 55.8 average, versus 34.0 event-level F1 for the training-free model improving to 47.5.
- Replacing temporal layers with linear layers drops unseen-class average from 55.8 to 28.3 (Table 5).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune the training-free baseline with seen-class training data and temporal layers | ImageBind OV-AVEL baseline · OV-AVEBench total test | Avg.: 46.6 -> 57.8 | Helped |
| fine-tune the training-free baseline with seen-class training data and temporal layers | ImageBind OV-AVEL baseline · OV-AVEBench seen test | Avg.: 45.5 -> 62.9 | Helped |
| fine-tune the training-free baseline with seen-class training data and temporal layers | ImageBind OV-AVEL baseline · OV-AVEBench unseen test | Avg.: 47.0 -> 55.8 | Helped |
| add the special text other (ablation) | Fine-tuning baseline · OV-AVEBench total test | Avg.: 47.0 -> 57.8 | Helped |
| add the special text other (ablation) | Fine-tuning baseline · OV-AVEBench seen test | Avg.: 49.7 -> 62.9 | Helped |
| add the special text other (ablation) | Fine-tuning baseline · OV-AVEBench unseen test | Avg.: 45.9 -> 55.8 | Helped |
| use geometric-mean fusion (Sqrt) instead of averaging probabilities (ablation) | Fine-tuning baseline · OV-AVEBench total test | Avg.: 39.0 -> 57.8 | Helped |
| use geometric-mean fusion (Sqrt) instead of averaging audio and visual features (ablation) | Fine-tuning baseline · OV-AVEBench total test | Avg.: 40.2 -> 57.8 | Helped |
| add temporal Transformer layers instead of segment-wise linear layers (ablation) | Fine-tuning baseline · OV-AVEBench total test | Avg.: 39.0 -> 57.8 | Helped |
| add temporal Transformer layers instead of segment-wise linear layers (ablation) | Fine-tuning baseline · OV-AVEBench seen test | Avg.: 65.8 -> 62.9 | Hurt |
| add temporal Transformer layers instead of segment-wise linear layers (ablation) | Fine-tuning baseline · OV-AVEBench unseen test | Avg.: 28.3 -> 55.8 | Helped |
| fine-tune using different training-data ratios (25%, 50%, 75%, or 100%) (ablation) | Fine-tuning baseline · OV-AVEBench total test | Avg.: 57.8; 57.6; 57.8; 57.7 | No clear effect |
Surprising findings
- Prior closed-set AVEL fusion modules (CMRA, AVE, PSP, MM-Pyramid) strongly improve seen-class performance but degrade unseen-class performance, sometimes to 31-39 average.
- Using only 25% of training data yields nearly the same average performance (57.7) as using 100% (57.8).
Limitations
- Training-free baseline performs poorly on seen classes relative to fine-tuned model.
- Unseen-class performance of closed-set fusion modules degrades strongly, indicating the task remains challenging.
- Dataset is built from VGGSound with 67 selected classes and background handling via an 'other' text class.
Experimental setup
Datasets: VGGSound OV-AVEBench
Benchmarks: OV-AVEBench
Metrics: accuracy segment-level F1 event-level F1 average of three metrics
Baselines: Video-LLaMA2 CLIP&CLAP CMRA AVE PSP MM-Pyramid
Builds on: ImageBind CLIP AVE OpenAVE
Closest papers by meaning
- Audio-Visual Semantic Graph Network for Audio-Visual Event Localization (CVPR 2025 · similarity 0.92)
- Dense Audio-Visual Event Localization Under Cross-Modal Consistency and Multi-Temporal Granularity Collaboration (AAAI 2025 · similarity 0.91)
- Adapting to the Unknown: Training-Free Audio-Visual Event Perception with Dynamic Thresholds (CVPR 2025 · similarity 0.90)
- PreFM: Online Audio-Visual Event Parsing via Predictive Future Modeling (NeurIPS 2025 · similarity 0.90)
- Empowering LLMs with Pseudo-Untrimmed Videos for Audio-Visual Temporal Understanding (AAAI 2025 · similarity 0.89)
- Multimodal Class-aware Semantic Enhancement Network for Audio-Visual Video Parsing (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.