AI Paper Atlas

Research map / Video / Video-language understanding

Weakly Supervised Video Representation Learning With Unaligned Text for Sequential Videos

Sixun Dong, Huazhang Hu, Dongze Lian, Weixin Luo, Yicheng Qian, Shenghua Gao

CVPR 2023 · Video › Video-language understanding · approach: Video-language pretraining · 18 citations (OpenAlex)

Read the paper

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add fine-grained frame-sentence contrastive loss with sort pseudo-labels (ablation)CLIP-ViT video representation · CSV (weakly supervised)AUC: 74.82 -> 79.80Improved
add coarse-grained video-paragraph contrastive loss (plus fine loss) (ablation)CLIP-ViT video representation · CSV (supervised w CLS)AUC: 83.58 -> 86.92 with both lossesImproved
add fine-grained loss only (ablation)CLIP-ViT video representation · CSV (supervised w CLS)AUC: 83.58 -> 84.85Improved
add coarse-grained loss only (ablation)CLIP-ViT video representation · CSV (supervised w CLS)AUC: 83.58 -> 84.32Improved
sort-based pseudo-label generation vs Viterbi vs splitting (ablation)fine-grained loss pseudo-labels · CSV (weakly supervised)AUC: sort 79.80, Viterbi 78.46, split 72.75Mixed
switch backbone from ResNet50 / ViT-B/32 to CLIP-ViT (ablation)video encoder · CSV (weakly supervised, fine loss)AUC: 78.32 (ResNet50) and 75.18 (ViT-B/32) -> 79.80 (CLIP-ViT)Improved
add Temporal Similarity Matrix (TSM) module to our method (ablation)Ours under weak supervision · CSVAUC: 79.80 -> 76.00Hurt
use paragraph feature by concatenating sentences vs temporal mean pooling (ablation)paragraph-level language representation · CSV (weakly supervised)AUC: 67.07 (pooling) -> 67.46 (concat)Improved
increase frames per video from 16 to 32 or 48 (batch 8) (ablation)Ours · CSV (weakly supervised)AUC: 67.46 (16) -> 65.64 (32), 64.40 (48); 8 frames 58.43Hurt
proposed multiple granularity loss vs baselines (CLIP+TE+MLP coarse only, CAT, MIL-NCE, VideoSwin)video representation · CSV, Diving-SV, COIN-SV (weakly supervised)AUC: CSV 74.82 -> 79.80; Diving-SV 81.47 -> 85.19; COIN-SV VideoSwin 54.73 beats ours 52.56Improved
add sequence alignment loss to supervised model (ablation)Ours (supervised) · CSVAUC: 84.47 -> 84.69Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy