Research map / Video / Video-language understanding
Adapting to the Unknown: Training-Free Audio-Visual Event Perception with Dynamic Thresholds
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AV2A, a training-free, open-vocabulary audio-visual event perception method that adds score-level fusion and within-video dynamic thresholds (label-shift estimation) on top of foundation models and existing weakly-supervised AVVP models.
Problem
Audio-visual event perception (temporal localization and classification of audio, visual, and audio-visual events) is limited to closed training vocabularies, relies on fixed thresholds that ignore temporal shifts in event distributions, and uses late fusion that loses multimodal interactions.
Approach
Replace late fusion with score-level fusion of audio and visual foundation-model scores, and replace static thresholds with per-video dynamic thresholds updated over time via a soft-confusion-matrix label-shift estimate scaled by cosine similarity of consecutive visual features, with no training.
Key results
- On LLP, AV2A with CLIP+CLAP improves segment-level Type@AV by 21.7% over the naive baseline, and with LanguageBind improves it by 30.4%.
- On AVE, AV2A with LanguageBind reaches 72.8% accuracy versus 64.7% for the DG-SCT zero-shot method.
- Applying dynamic thresholds to MGN-MA, JoMoLD and CMPAE gives consistent small gains on LLP without any extra training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply AV2A to CLIP + CLAP | CLIP + CLAP · LLP | Type@AV segment: 22.1 -> 43.8 ± .04 | Helped |
| Apply AV2A to CLIP + CLAP | CLIP + CLAP · LLP | Type@AV event: 19.5 -> 39.4 ± .01 | Helped |
| Apply AV2A to LanguageBind | LanguageBind · LLP | Type@AV segment: 22.0 -> 52.4 ± .04 | Helped |
| Apply AV2A to LanguageBind | LanguageBind · LLP | Type@AV event: 19.5 -> 47.5 ± .04 | Helped |
| Apply AV2A dynamic thresholds | CMPAE · LLP | Type@AV segment: 63.3 -> 63.6 | Helped |
| Apply AV2A dynamic thresholds | CMPAE · LLP | Type@AV event: 57.7 -> 58.0 | Helped |
| Apply AV2A to CLIP + CLAP | CLIP + CLAP · AVE | accuracy: 38.2 ± 2.87 -> 68.5 ± 0.16 | Helped |
| Apply AV2A to LanguageBind | LanguageBind · AVE | accuracy: 32.9 ± 1.27 -> 72.8 ± 0.14 | Helped |
| Add cosine similarity to label shift (ablation) | AV2A (LanguageBind) · LLP | Type@AV segment: 52.3 -> 52.4 | No clear effect |
| Add cosine similarity to label shift (ablation) | AV2A (LanguageBind) · LLP | Type@AV event: 47.3 -> 47.5 | No clear effect |
| Use dynamic rather than fixed thresholds (ablation) | AV2A (LanguageBind) · LLP | Type@AV segment: 52.2 -> 52.4 | Helped |
| Use dynamic rather than fixed thresholds (ablation) | AV2A (LanguageBind) · LLP | Type@AV event: 47.2 -> 47.5 | Helped |
Limitations
- Reliance on foundation models may introduce biases and limit accuracy on short-duration or rapid events.
- Hyperparameters are tuned on a subset of the LLP validation set, so transfer to other settings is uncertain.
Experimental setup
Datasets: AVE LLP
Benchmarks: AVE LLP
Metrics: F1-score (segment and event level) Type@AV Event@AV accuracy
Baselines: CLIP+CLAP training-free baseline LanguageBind training-free baseline DG-SCT MGN-MA JoMoLD CMPAE
Builds on: BBSE label shift estimation CLIP CLAP LanguageBind multi-modal multiple instance learning
Closest papers by meaning
- Dense Audio-Visual Event Localization Under Cross-Modal Consistency and Multi-Temporal Granularity Collaboration (AAAI 2025 · similarity 0.90)
- Towards Open-Vocabulary Audio-Visual Event Localization (CVPR 2025 · similarity 0.90)
- PreFM: Online Audio-Visual Event Parsing via Predictive Future Modeling (NeurIPS 2025 · similarity 0.90)
- Empowering LLMs with Pseudo-Untrimmed Videos for Audio-Visual Temporal Understanding (AAAI 2025 · similarity 0.89)
- From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation (ICML 2024 · similarity 0.88)
- LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videos (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.