Research map / Video / Video segmentation and tracking
UWAV: Uncertainty-weighted Weakly-supervised Audio-Visual Video Parsing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UWAV, a weakly-supervised audio-visual video parsing method that pre-trains transformer-based CLIP/CLAP pseudo-label generators on UnAV, then trains a HAN inference model with uncertainty-weighted pseudo-labels, feature mixup, and class-balanced re-weighting.
Problem
Audio-visual video parsing requires localizing uni-modal and multi-modal events per segment, but only video-level labels are available during training, and existing segment-level pseudo-labels ignore inter-segment dependencies and uncertainty and suffer from class imbalance.
Approach
Temporally-aware transformers on frozen CLIP/CLAP features, pre-trained on the large UnAV dataset, generate temporally coherent segment pseudo-labels; the pseudo-label confidence (distance of logits from thresholds) weights the training loss, feature mixup regularizes training with interpolated labels, and class-frequency re-weighting handles negative/positive imbalance.
Key results
- On LLP, UWAV reaches 65.9 segment-level Type@AV F1 versus 64.8 for the prior best PPL.
- On AVE, UWAV achieves 80.6% accuracy versus 80.4% for VALOR and 75.3% for HAN.
- Ablation shows uncertainty-weighted pseudo-labels improve segment-level Type@AV by about 2 points over binary pseudo-labels.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use UWAV instead of VALOR | AVE weakly supervised video parsing · AVE | accuracy: 80.4 -> 80.6 | Helped |
| Use UWAV instead of PPL | LLP video parsing · LLP segment-level | Type@AV F1: 64.8 -> 65.9 | Helped |
| Use UWAV instead of PPL | LLP video parsing · LLP event-level | Type@AV F1: 59.9 -> 60.9 | Helped |
| Generate pseudo-labels with the temporally dependent UWAV modules instead of VALOR | LLP pseudo-label generation · LLP segment-level | Type@AV: 66.0 -> 72.8 | Helped |
| Use uncertainty-weighted pseudo-labels instead of binary pseudo-labels (ablation) | UWAV inference model · LLP segment-level | Type@AV F1: 64.2 -> 64.4 | Helped |
| Add class-balanced loss re-weighting (ablation) | UWAV with uncertainty-weighted pseudo-labels · LLP segment-level | Type@AV F1: 64.4 -> 65.4 | Helped |
| Add uncertainty-weighted feature mixup (ablation) | UWAV with uncertainty-weighted pseudo-labels · LLP segment-level | Type@AV F1: 64.4 -> 65.2 | Helped |
Limitations
- Gains over the closest competitor on AVE are small (80.6 vs 80.4)
- Relies on a large supervised pre-training dataset (UnAV) for the pseudo-label generators
- Requires pre-defined class-wise thresholds for pseudo-label generation
Experimental setup
Datasets: LLP AVE UnAV
Benchmarks: LLP AVE
Metrics: segment-level F1 (A, V, AV, Type@AV, Event@AV) event-level F1 accuracy
Baselines: HAN MA JoMoLD CMPAE PoiBin VPLAN VALOR LSLD PPL CoLeaf LEAP
Builds on: HAN CLIP CLAP MMIL pooling mixup
Closest papers by meaning
- Weakly-Supervised Audio-Visual Video Parsing with Prototype-based Pseudo-Labeling (CVPR 2024 · similarity 0.94)
- Multimodal Class-aware Semantic Enhancement Network for Audio-Visual Video Parsing (AAAI 2025 · similarity 0.88)
- PreFM: Online Audio-Visual Event Parsing via Predictive Future Modeling (NeurIPS 2025 · similarity 0.87)
- Empowering LLMs with Pseudo-Untrimmed Videos for Audio-Visual Temporal Understanding (AAAI 2025 · similarity 0.86)
- video-SALMONN: Speech-Enhanced Audio-Visual Large Language Models (ICML 2024 · similarity 0.86)
- VSRo-200: A Romanian Visual Speech Recognition Dataset for Studying Supervision and Multimodal Robustness (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.