Research map / Video / Video-language understanding
Stitching Segments and Sentences towards Generalization in Video-Text Pre-training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes S3VL, a video-language pre-training method that stitches video frames or sentences into longer sequences and predicts fine-grained segment boundaries/positions, improving zero-shot and fine-tuned retrieval, QA, captioning and moment retrieval.
Problem
Video-language pre-training with contrastive learning and masked language modeling aligns only global features, missing fine-grained frame-word associations needed for matching and temporal tasks like moment retrieval in long videos.
Approach
A stitching-and-matching pre-text task that concatenates frame tokens from different videos (or sentence tokens/[CLS] tokens from different sentences) and trains the multimodal encoder to predict the start/end boundary of the query-matched segment or the matching sentence, requiring no fine-grained annotations.
Key results
- Zero-shot text-to-video retrieval R@1 on MSR-VTT 27.2 (vs. Clover 25.8), DiDeMo 31.3 and LSMDC 14.9 under W2M+CC3M pre-training.
- Fine-tuned MSR-VTT text-to-video R@1 of 41.0 vs. Clover 38.6; MSR-VTT VQA accuracy 44.7 and MSVD captioning CIDEr 61.9 (open-ended setting).
- Moment retrieval on Charades-STA Rank@0.5 of 49.6 vs. UMT 48.3 and LocVTP 41.2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add stitching-and-matching pre-training to contrastive learning and MLM (ablation) | S3VL with SwinT · MSR-VTT zero-shot retrieval (WebVid1M ablation) | Recall@1: 21.2 -> 22.4 | Helped |
| Add stitching video and matching text (Lsvmt) (ablation) | S3VL with SwinT · MSR-VTT zero-shot retrieval (WebVid1M ablation) | Recall@1: 21.2 -> 22.1 | Helped |
| Add stitching text and matching video (Lstmv) (ablation) | S3VL with SwinT · MSR-VTT zero-shot retrieval (WebVid1M ablation) | Recall@1: 21.2 -> 21.9 | Helped |
| Use hard sampling of frame features from the most similar videos in a batch (ablation) | S3VL video stitching · MSR-VTT retrieval | Recall@1: 21.7 (Shuffling), 21.5 (Sampling), 22.1 (HardSampling) | Helped |
| Stitch only sentence [CLS] tokens instead of stitching words (ablation) | S3VL sentence stitching · MSR-VTT retrieval | Recall@1: 21.7 -> 21.9 | Helped |
| Use boundary classification instead of boundary regression or masking classification (ablation) | S3VL boundary prediction · Downstream tasks shown in Figure 3 | Performance | Helped |
| Add stitching and matching pre-training (ablation) | ViT video backbone · MSR-VTT retrieval | Recall@1: 16.4 -> 17.7 | Helped |
| Add stitching and matching pre-training (ablation) | SwinT video backbone · MSR-VTT retrieval | Recall@1: 20.8 -> 22.4 | Helped |
| Use S3VL instead of Lavender | Video-language model · MSVD VQA | Accuracy: 55.4 -> 53.9 | Hurt |
| Use S3VL instead of UMT | Moment retrieval model · Charades-STA | Rank@0.5: 48.3 -> 49.6 | Helped |
| Use S3VL instead of UMT | Moment retrieval model · Charades-STA | Rank@0.7: 29.3 -> 28.8 | Hurt |
Limitations
- Moment retrieval uses off-the-shelf algorithms on extracted features rather than end-to-end training
- Pre-training data is limited to WebVid and CC3M, mostly short videos
- Rank@0.7 is lower than UMT on Charades-STA
Experimental setup
Datasets: WebVid Google Conceptual Captions Kinetics-400 MSR-VTT DiDeMo LSMDC MSVD Charades-STA
Benchmarks: MSR-VTT DiDeMo LSMDC MSVD Charades-STA
Metrics: Recall@1 Recall@5 Recall@10 accuracy CIDEr Rank@0.5 Rank@0.7 AVG
Baselines: Frozen VIOLET ALPRO LocVTP MCQ OA-Trans Miles Clover RegionLearner HD-VILA All-in-one Lavender
Builds on: ALPRO Clover VideoSwin BERT
Closest papers by meaning
- Towards Balanced Alignment: Modal-Enhanced Semantic Modeling for Video Moment Retrieval (AAAI 2024 · similarity 0.88)
- TOPA: Extending Large Language Models for Video Understanding via Text-Only Pre-Alignment (NeurIPS 2024 · similarity 0.88)
- Frame-Voyager: Learning to Query Frames for Video Large Language Models (ICLR 2025 · similarity 0.87)
- VideoComp: Advancing Fine-Grained Compositional and Temporal Alignment in Video-Text Models (CVPR 2025 · similarity 0.87)
- LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment (ICLR 2024 · similarity 0.87)
- Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.