Research map / Video / Video-language understanding
Weakly Supervised Video Representation Learning With Unaligned Text for Sequential Videos
Sixun Dong, Huazhang Hu, Dongze Lian, Weixin Luo, Yicheng Qian, Shenghua Gao
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add fine-grained frame-sentence contrastive loss with sort pseudo-labels (ablation) | CLIP-ViT video representation · CSV (weakly supervised) | AUC: 74.82 -> 79.80 | Improved |
| add coarse-grained video-paragraph contrastive loss (plus fine loss) (ablation) | CLIP-ViT video representation · CSV (supervised w CLS) | AUC: 83.58 -> 86.92 with both losses | Improved |
| add fine-grained loss only (ablation) | CLIP-ViT video representation · CSV (supervised w CLS) | AUC: 83.58 -> 84.85 | Improved |
| add coarse-grained loss only (ablation) | CLIP-ViT video representation · CSV (supervised w CLS) | AUC: 83.58 -> 84.32 | Improved |
| sort-based pseudo-label generation vs Viterbi vs splitting (ablation) | fine-grained loss pseudo-labels · CSV (weakly supervised) | AUC: sort 79.80, Viterbi 78.46, split 72.75 | Mixed |
| switch backbone from ResNet50 / ViT-B/32 to CLIP-ViT (ablation) | video encoder · CSV (weakly supervised, fine loss) | AUC: 78.32 (ResNet50) and 75.18 (ViT-B/32) -> 79.80 (CLIP-ViT) | Improved |
| add Temporal Similarity Matrix (TSM) module to our method (ablation) | Ours under weak supervision · CSV | AUC: 79.80 -> 76.00 | Hurt |
| use paragraph feature by concatenating sentences vs temporal mean pooling (ablation) | paragraph-level language representation · CSV (weakly supervised) | AUC: 67.07 (pooling) -> 67.46 (concat) | Improved |
| increase frames per video from 16 to 32 or 48 (batch 8) (ablation) | Ours · CSV (weakly supervised) | AUC: 67.46 (16) -> 65.64 (32), 64.40 (48); 8 frames 58.43 | Hurt |
| proposed multiple granularity loss vs baselines (CLIP+TE+MLP coarse only, CAT, MIL-NCE, VideoSwin) | video representation · CSV, Diving-SV, COIN-SV (weakly supervised) | AUC: CSV 74.82 -> 79.80; Diving-SV 81.47 -> 85.19; COIN-SV VideoSwin 54.73 beats ours 52.56 | Improved |
| add sequence alignment loss to supervised model (ablation) | Ours (supervised) · CSV | AUC: 84.47 -> 84.69 | Improved |
Closest papers by meaning
- TOPA: Extending Large Language Models for Video Understanding via Text-Only Pre-Alignment (NeurIPS 2024 · similarity 0.88)
- Enhanced Motion-Text Alignment for Image-to-Video Transfer Learning (CVPR 2024 · similarity 0.87)
- Bridging the Semantic Granularity Gap Between Text and Frame Representations for Partially Relevant Video Retrieval (AAAI 2025 · similarity 0.87)
- OST: Refining Text Knowledge with Optimal Spatio-Temporal Descriptor for General Video Recognition (CVPR 2024 · similarity 0.86)
- VideoPrism: A Foundational Visual Encoder for Video Understanding (ICML 2024 · similarity 0.86)
- A Simple Recipe for Contrastively Pre-training Video-First Encoders Beyond 16 Frames (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
