Research map / Video / Video segmentation and tracking
Weakly-Supervised Audio-Visual Video Parsing with Prototype-based Pseudo-Labeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes prototype-based pseudo-labeling with contrastive learning and an EM interpretation to improve weakly-supervised audio-visual video parsing on LLP, built on the HAN architecture.
Problem
Weakly-supervised audio-visual video parsing (AVVP) must predict segment-level audio/visual event labels while training only with video-level labels; MIL models tend to focus on the most discriminative segments and miss full event extent.
Approach
Cluster key segments identified by a MIL model into multiple per-class prototypes (k-means on positive and negative segment features from a momentum encoder), assign segment-level pseudo labels by feature similarity to these prototypes, retrain with pseudo-label BCE plus pseudo-label contrastive loss, and interpret the iterative procedure as EM.
Key results
- Achieves Type@AV 63.7 segment-level and Event@AV 57.9 (vs. CMPAE 62.8 and 55.7 respectively) on LLP, a 1.86-point average improvement over CMPAE across all metrics.
- Audio-visual segment-level F1 reaches 61.9 vs. 59.2 for CMPAE-level previous best in the table comparison (CMPAE 59.2), with a 2.6-point average gain on audio-visual events.
- On THUMOS14 weakly-supervised TAL, average mAP over tIoU 0.1:0.7 is 46.9, matching DELU (46.4) and outperforming it on tIoU 0.7 (20.1 vs 15.3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace MIL segment predictions with naive hard pseudo labels (ablation) | Base MIL model on LLP · LLP test | Type@AV segment F1: 57.6 -> 57.8 | No clear effect |
| replace MIL segment predictions with naive soft pseudo labels (ablation) | Base MIL model on LLP · LLP test | Type@AV segment F1: 57.6 -> 58.2 | No clear effect |
| use prototype-based hard pseudo labels (ablation) | Base MIL model on LLP · LLP test | Type@AV segment F1: 57.6 -> 61.9 | Helped |
| use prototype-based soft rather than hard pseudo labels (ablation) | Base MIL model with prototype pseudo-labeling on LLP · LLP test | Type@AV segment F1: 61.9 -> 64.8 | Helped |
| add contrastive learning based on pseudo labels (ablation) | Prototype pseudo-labeling model on LLP · LLP test | Type@AV segment F1: 61.2 -> 64.8 | Helped |
| increase the number of prototypes from too few to 10 per class (ablation) | Prototype pseudo-labeling model · LLP test | average segment/event F1 | Helped |
| use too few prototypes per class (ablation) | Prototype pseudo-labeling model · LLP test | average segment/event F1 | Hurt |
| vary positive-segment threshold γ (ablation) | Prototype pseudo-labeling model · LLP test | average segment/event F1 | Mixed |
| vary negative-segment threshold β (ablation) | Prototype pseudo-labeling model · LLP test | average segment/event F1 | No clear effect |
Surprising findings
- Naive pseudo-labeling using the MIL model's own segment predictions gives no consistent improvement over the baseline.
- Hard pseudo labels underperform soft pseudo labels, and performance is sensitive to the negative-selection hyperparameter beta.
Limitations
- Performance is sensitive to the negative-selection hyperparameter beta.
- Same number of prototypes is assumed for all events despite differing intra-class variation.
- EM may converge only to a local optimum.
Experimental setup
Datasets: LLP THUMOS14
Benchmarks: LLP THUMOS14
Metrics: segment-level F1 event-level F1 Type@AV Event@AV mAP at tIoU
Baselines: HAN MA CVCM-MA MGN-MA JoMoLD PoiBin CMPAE EM-MIL ASM-Loc DELU
Builds on: HAN (Hybrid Attention Network) multiple-instance learning momentum encoder (MoCo) Look, Listen and Parse (LLP)
Closest papers by meaning
- Revisit Weakly-Supervised Audio-Visual Video Parsing from the Language Perspective (NeurIPS 2023 · similarity 0.94)
- UWAV: Uncertainty-weighted Weakly-supervised Audio-Visual Video Parsing (CVPR 2025 · similarity 0.94)
- Multimodal Class-aware Semantic Enhancement Network for Audio-Visual Video Parsing (AAAI 2025 · similarity 0.87)
- PreFM: Online Audio-Visual Event Parsing via Predictive Future Modeling (NeurIPS 2025 · similarity 0.86)
- Empowering LLMs with Pseudo-Untrimmed Videos for Audio-Visual Temporal Understanding (AAAI 2025 · similarity 0.85)
- Adapting to the Unknown: Training-Free Audio-Visual Event Perception with Dynamic Thresholds (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.