Research map / Vision-language and multimodal / Cross-modal learning
CoLeaF: A Contrastive-Collaborative Learning Framework for Weakly Supervised Audio-Visual Video Parsing
Faegheh Sardari, Armin Mustafa, Philip JB Jackson, Adrian Hilton
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace event-aware contrastive loss with standard NCE loss (ablation) | CoLeaF · LLP (segment-level) | Ao F-score: 47.3 (no loss) -> 46.1 (NCE) | Hurt |
| add event-aware contrastive loss L_Evt (ablation) | CoLeaF · LLP (segment-level) | Ao F-score: 47.3 -> 49.3 | Improved |
| use BCE on pseudo-labels instead of event-aware contrastive loss (ablation) | CoLeaF · LLP (segment-level) | Vo F-score: 61.8 (no loss) -> 61.2 (BCE); Ao 47.3 -> 48.5 | Hurt |
| add self-modality-aware knowledge distillation (ablation) | CoLeaF · LLP (segment-level) | Vo F-score: 60.9 -> 62.4 | Improved |
| add both class tokens and co-occurrence class distillation loss (ablation) | CoLeaF · LLP (segment-level) | AV F-score: 56.6 -> 58.6 | Improved |
| add class tokens without the co-occurrence distillation loss (ablation) | CoLeaF · LLP (segment-level) | Vo F-score: 60.8 -> 60.3 | No clear effect |
| add co-occurrence class distillation loss without class tokens (ablation) | CoLeaF · LLP (segment-level) | Vo F-score: 60.8 -> 62.7 | Improved |
| embed HAN in CoLeaF as the anchor branch | HAN · LLP (segment-level) | Ao F-score: 33.1 -> 36.3 | Improved |
| embed CMPAE in CoLeaF as the anchor branch | CMPAE · LLP (segment-level) | Vo F-score: 57.9 -> 62.4 | Improved |
| use CoLeaF instead of CMPAE on traditional A metric | LLP · LLP (segment-level, standard threshold) | A F-score: 63.9 -> 63.5 | No clear effect |
| train with extra CLIP and CLAP pseudo-labels | CoLeaF vs prior methods · LLP (segment-level) | Ao F-score: 49.0 -> 51.2 over VALOR | Improved |
| use CoLeaF on UnAV-100 weakly supervised localization | CoLeaF · UnAV-100 | AV F-score: +1.8% segment, +3.0% event | Improved |
Closest papers by meaning
- Exploring Cross-Video and Cross-Modality Signals for Weakly-Supervised Audio-Visual Video Parsing (NeurIPS 2021 · similarity 0.94)
- Weakly-Supervised Audio-Visual Video Parsing with Prototype-based Pseudo-Labeling (CVPR 2024 · similarity 0.92)
- UWAV: Uncertainty-weighted Weakly-supervised Audio-Visual Video Parsing (CVPR 2025 · similarity 0.92)
- Multimodal Class-aware Semantic Enhancement Network for Audio-Visual Video Parsing (AAAI 2025 · similarity 0.90)
- PreFM: Online Audio-Visual Event Parsing via Predictive Future Modeling (NeurIPS 2025 · similarity 0.88)
- LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videos (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
