Research map / Video / Video-language understanding
SRTube: Video-Language Pre-Training with Action-Centric Video Tube Features and Semantic Role Labeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SRTube, a video-language pre-training framework that combines object-trajectory tube features with semantic role labeling (SRL) phrase features and two action-centric proxy tasks (MAM, ANM) to improve cross-modal alignment.
Problem
Existing video-language pre-training models capture temporal dynamics and object-level alignment poorly and ignore action-related semantic structure in language, limiting cross-modal alignment for action-centric video tasks.
Approach
Build a tube builder (pre-trained on AVA v2 action recognition) that yields multiple object-trajectory tube features, pair them with SRL-derived verb/argument semantic phrase features via a tube-to-SRL attention in a cross-fusion transformer, and add masked action modeling and action numbering modeling objectives.
Key results
- On MSR-VTT fine-tuned text-to-video retrieval, SRTube reaches R@1 46.3 vs. VindLU 43.8 with the same 5.5M pre-training pairs.
- Zero-shot MSR-VTT R@1 improves to 29.1 from BridgeFormer's 26.0 (about 3.2% gain).
- Video captioning on MSR-VTT reaches CIDEr 61.8, above STOA-VLP's 60.2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add tube features to baseline visual and text features (ablation) | SRTube · MSR-VTT zero-shot text-to-video retrieval | R@1: 24.7 -> 26.3 | Helped |
| Add SRL semantic phrase features to baseline visual and text features (ablation) | SRTube · MSR-VTT zero-shot text-to-video retrieval | R@1: 24.7 -> 26.1 | Helped |
| Add both tube and SRL semantic phrase features (ablation) | SRTube · MSR-VTT zero-shot text-to-video retrieval | R@1: 24.7 -> 28.5 | Helped |
| Add MAM objective (ablation) | SRTube pre-training · MSVD VQA | accuracy: 44.7 -> 44.9 | Helped |
| Add ANM objective (ablation) | SRTube pre-training · MSVD VQA | accuracy: 44.7 -> 45.4 | Helped |
| Add MAM and ANM objectives (ablation) | SRTube pre-training · MSVD VQA | accuracy: 44.7 -> 46.0 | Helped |
| Use 16 rather than 4 frames (ablation) | SRTube tube builder · AVA v2 | mAP: 24.1 -> 26.1 | Helped |
| Increase frame count from 16 to 32 (ablation) | SRTube tube builder · AVA v2 | mAP: 26.1 -> 24.9 | Hurt |
| Use 8 rather than 4 tube queries (ablation) | SRTube tube builder · AVA v2 | mAP: 23.0 -> 26.2 | Helped |
| Increase tube queries from 8 to 12 (ablation) | SRTube tube builder · AVA v2 | mAP: 26.2 -> 24.4 | Hurt |
| Increase tube queries from 8 to 16 (ablation) | SRTube tube builder · AVA v2 | mAP: 26.2 -> 24.5 | Hurt |
| Use SRTube instead of VindLU | VidLP model · MSR-VTT text-to-video retrieval | R@1: 43.8 -> 46.3 | Helped |
Surprising findings
- Adding the SRL phrase feature alone slightly lowers R@10 on MSR-VTT ablation (51.8 vs 52.1 baseline), which the authors attribute to SRL over-recognizing verbs.
- On DiDeMo, SRTube is second-ranked at R@5 and R@10, slightly behind VindLU, which the authors attribute to long videos with scene changes.
Limitations
- Tube features assume continuous scenes, which hurts on long videos with scene changes such as DiDeMo.
- SRL can over-recognize verbs, causing slight drops in some metrics.
- Adaptive frame selection is not addressed.
Experimental setup
Datasets: WebVid2M CC3M AVA v2 MSR-VTT DiDeMo LSMDC SSv2 ActivityNet Captions MSVD-QA TVQA MSVD
Benchmarks: MSR-VTT LSMDC DiDeMo SSv2 ActivityNet Captions MSRVTT-QA MSVD-QA TVQA MSVD AVA v2
Metrics: R@1 R@5 R@10 accuracy CIDEr mAP
Baselines: VindLU BridgeFormer OA-Trans HiTeA TW-BERT ALPRO Singularity X-CLIP STOA-VLP LAVENDER
Builds on: VindLU BEiT Deformable DETR-style tube queries (Yang et al. action tube) BridgeFormer OA-Trans
Closest papers by meaning
- Structured Video-Language Modeling with Temporal Grouping and Spatial Grounding (ICLR 2024 · similarity 0.87)
- Align Before Adapt: Leveraging Entity-to-Region Alignments for Generalizable Video Action Recognition (CVPR 2024 · similarity 0.87)
- Kronecker Mask and Interpretive Prompts are Language-Action Video Learners (ICLR 2025 · similarity 0.86)
- VidLA: Video-Language Alignment at Scale (CVPR 2024 · similarity 0.86)
- LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment (ICLR 2024 · similarity 0.86)
- Weakly Supervised Temporal Action Localization via Dual-Prior Collaborative Learning Guided by Multimodal Large Language Models (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.