AI Paper Atlas

Research map / Vision-language and multimodal / Cross-modal learning

CoLeaF: A Contrastive-Collaborative Learning Framework for Weakly Supervised Audio-Visual Video Parsing

Faegheh Sardari, Armin Mustafa, Philip JB Jackson, Adrian Hilton

ECCV 2024 · Vision-language and multimodal › Cross-modal learning · approach: Self-supervised audio-visual representation learning · 9 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
replace event-aware contrastive loss with standard NCE loss (ablation)CoLeaF · LLP (segment-level)Ao F-score: 47.3 (no loss) -> 46.1 (NCE)Hurt
add event-aware contrastive loss L_Evt (ablation)CoLeaF · LLP (segment-level)Ao F-score: 47.3 -> 49.3Improved
use BCE on pseudo-labels instead of event-aware contrastive loss (ablation)CoLeaF · LLP (segment-level)Vo F-score: 61.8 (no loss) -> 61.2 (BCE); Ao 47.3 -> 48.5Hurt
add self-modality-aware knowledge distillation (ablation)CoLeaF · LLP (segment-level)Vo F-score: 60.9 -> 62.4Improved
add both class tokens and co-occurrence class distillation loss (ablation)CoLeaF · LLP (segment-level)AV F-score: 56.6 -> 58.6Improved
add class tokens without the co-occurrence distillation loss (ablation)CoLeaF · LLP (segment-level)Vo F-score: 60.8 -> 60.3No clear effect
add co-occurrence class distillation loss without class tokens (ablation)CoLeaF · LLP (segment-level)Vo F-score: 60.8 -> 62.7Improved
embed HAN in CoLeaF as the anchor branchHAN · LLP (segment-level)Ao F-score: 33.1 -> 36.3Improved
embed CMPAE in CoLeaF as the anchor branchCMPAE · LLP (segment-level)Vo F-score: 57.9 -> 62.4Improved
use CoLeaF instead of CMPAE on traditional A metricLLP · LLP (segment-level, standard threshold)A F-score: 63.9 -> 63.5No clear effect
train with extra CLIP and CLAP pseudo-labelsCoLeaF vs prior methods · LLP (segment-level)Ao F-score: 49.0 -> 51.2 over VALORImproved
use CoLeaF on UnAV-100 weakly supervised localizationCoLeaF · UnAV-100AV F-score: +1.8% segment, +3.0% eventImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy