Research map / Video / Video-language understanding
Structured Video-Language Modeling with Temporal Grouping and Spatial Grounding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes S-ViLM, a dual-encoder video-language pre-training framework that adds cut-and-paste temporal grouping and group-token spatial grounding losses to global contrastive learning.
Problem
Video-language pre-training mainly aligns holistic video-caption pairs and neglects fine-grained region-object correspondences and temporal scene changes, which limits localization and reasoning tasks.
Approach
Use cut-and-paste to synthesize scene changes, a temporal grouping loss that separates foreground and background clips, and learnable group tokens that cluster video regions and ground them to noun phrases from captions, all trained jointly with global contrastive loss.
Key results
- Zero-shot MSR-VTT text-video retrieval R@1 of 28.6 and R@10 of 65.1 with VideoCC+ActivityNet pre-training, versus 26.0 R@1 for MCQ.
- UCF101 linear probing accuracy 94.8 and HMDB51 linear probing 70.0, above MCQ (89.1 and 65.8).
- TAL mAP@0.5 of 51.7 with G-TAD on ActivityNet using frozen features, exceeding BSP and LoFi.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add spatial grounding loss to contrastive learning (ablation) | S-ViLM · MSR-VTT zero-shot | R@1: 22.7 -> 23.3 | Helped |
| Add spatial grounding loss to contrastive learning (ablation) | S-ViLM · MSVD-QA | accuracy: 43.6 -> 44.1 | Helped |
| Add temporal grouping loss to contrastive learning (ablation) | S-ViLM · MSR-VTT zero-shot | R@1: 22.7 -> 24.2 | Helped |
| Add temporal grouping loss to contrastive learning (ablation) | S-ViLM · MSVD-QA | accuracy: 43.6 -> 43.9 | Helped |
| Add spatial grounding and temporal grouping (ablation) | S-ViLM · MSR-VTT zero-shot | R@1: 22.7 -> 24.7 | Helped |
| Add spatial grounding and temporal grouping (ablation) | S-ViLM · MSVD-QA | accuracy: 43.6 -> 44.9 | Helped |
| Add spatial grounding and temporal grouping (ablation) | S-ViLM · UCF101 | accuracy: 90.5 -> 91.0 | Helped |
| Pre-train on VideoCC and ActivityNet rather than VideoCC alone (ablation) | S-ViLM · MSR-VTT zero-shot | R@1: 24.7 -> 28.6 | Helped |
| Pre-train on VideoCC and ActivityNet rather than VideoCC alone (ablation) | S-ViLM · ActivityNet TAL | average mAP: 34.2 -> 34.7 | Helped |
| Use HowTo100M instead of VideoCC for pre-training (ablation) | S-ViLM · MSR-VTT zero-shot | R@1: 24.7 -> 9.4 | Hurt |
| Use HowTo100M instead of VideoCC and ActivityNet for pre-training (ablation) | S-ViLM · ActivityNet TAL | average mAP: 34.7 -> 35.6 | Helped |
| Increase input from 16 frames to 32 frames (ablation) | S-ViLM · MSR-VTT zero-shot | R@10: 64.0 -> 65.1 | Helped |
Surprising findings
- Pre-training on HowTo100M alone gave the best TAL result (avg 35.6), slightly above VideoCC+ActivityNet (34.7), while the same setup underperforms on retrieval.
- HowTo100M pre-training yields much weaker zero-shot retrieval (R@1 9.4) than VideoCC (24.7) for the same method.
Limitations
- Pre-training uses VideoCC and ActivityNet-Captions rather than the commonly used WebVid, limiting direct comparison with baselines.
- Only the video encoder is used for action recognition and TAL, and TAL relies on frozen features with G-TAD rather than end-to-end training.
Experimental setup
Datasets: VideoCC ActivityNet-Captions HowTo100M WebVid MSR-VTT MSRVTT-QA MSVD-QA HMDB51 UCF101 ActivityNet AVA v2.2
Benchmarks: MSR-VTT MSRVTT-QA MSVD-QA HMDB51 UCF101 ActivityNet AVA v2.2
Metrics: R@1 R@5 R@10 MedR top-1 accuracy mAP at tIoU thresholds
Baselines: VCC MCQ ALPRO DemoVLP Frozen VIOLETv2 All-in-One MMV TSP BSP PAL
Builds on: CLIP-style contrastive learning grouping blocks (GroupViT) ALPRO MCQ
Closest papers by meaning
- Weakly Supervised Temporal Action Localization via Dual-Prior Collaborative Learning Guided by Multimodal Large Language Models (CVPR 2025 · similarity 0.89)
- Universal Video Temporal Grounding with Generative Multi-modal Large Language Models (NeurIPS 2025 · similarity 0.88)
- VidLA: Video-Language Alignment at Scale (CVPR 2024 · similarity 0.88)
- Unleashing the Potential of Multimodal LLMs for Zero-Shot Spatio-Temporal Video Grounding (NeurIPS 2025 · similarity 0.88)
- Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoning (ICML 2024 · similarity 0.87)
- ReVisionLLM: Recursive Vision-Language Model for Temporal Grounding in Hour-Long Videos (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.