Research map / Video / Video-language understanding
Video Language Model Pretraining with Spatio-temporal Masking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes spatio-temporal masking (STM) with a text-guided spatio-temporal decoder for self-supervised video-language pretraining, reconstructing masked video features from an ActionCLIP teacher.
Problem
Mask-based video-language pretraining reconstructs image features better than video features downstream, which the authors attribute to inconsistency between masking strategy and reconstruction target.
Approach
Mask visual tokens with high spatio-temporal contribution (spatial attention plus predicted temporal points across adjacent frames) so that masked targets match spatio-temporal feature reconstruction, and decode the masked tokens with text-guided semantic enhancement.
Key results
- STM reaches 48.5 R@1 on MSRVTT text-to-video retrieval with 5M pretraining pairs, up 2.2 points over UMT (46.3) under the same data.
- STM achieves 63.9 CIDEr on MSRVTT and 145.4 on MSVD captioning, outperforming HiTeA (62.5 and 145.1) with 5M pairs.
- Spatio-temporal masking gives 39.7 retrieval vs 37.4 for spatial masking on MSRVTT ablation, and a 30% masking ratio works best.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use STM instead of UMT | Video-language pretraining on WebVid-2M and CC3M · MSRVTT | Recall@1: 46.3 -> 48.5 | Helped |
| use STM instead of UMT | Video-language pretraining on WebVid-2M and CC3M · DiDeMo | Recall@1: 54.8 -> 56.9 | Helped |
| use STM instead of UMT | Video-language pretraining on WebVid-2M and CC3M · ActivityNet | Recall@1: 52.1 -> 53.6 | Helped |
| use STM instead of HiTeA | Video captioning · MSRVTT | CIDEr: 62.5 -> 63.9 | Helped |
| use STM instead of HiTeA | Video captioning · MSVD | CIDEr: 145.1 -> 145.4 | Helped |
| replace spatial-only masking with spatio-temporal masking (ablation) | STM · MSRVTT | retrieval score: 37.4 -> 39.7 | Helped |
| replace spatial-only masking with spatio-temporal masking (ablation) | STM · MSRVTT | QA score: 39.0 -> 41.2 | Helped |
| replace spatial-only masking with spatio-temporal masking (ablation) | STM · MSRVTT | captioning score: 53.5 -> 55.9 | Helped |
| add text guidance to the spatio-temporal decoder (ablation) | STM · MSRVTT | QA score: 39.0 -> 41.2 | Helped |
| add masked visual modeling loss (ablation) | STM with VTM, VTC and MLM losses · MSRVTT | retrieval score: 36.1 -> 39.7 | Helped |
| add masked visual modeling loss (ablation) | STM with VTM, VTC and MLM losses · MSRVTT | QA score: 38.2 -> 41.2 | Helped |
| use spatio-temporal rather than spatial reconstruction targets with spatio-temporal masking (ablation) | STM · MSRVTT | retrieval score: 36.7 -> 39.7 | Helped |
Surprising findings
- Spatio-temporal masking hurts when reconstructing image (spatial) features, and matching the masked target to the reconstruction target is what matters.
- A stronger video teacher (VideoMAEv2) gave lower performance than ActionCLIP, suggesting multimodal alignment matters more than action recognition ability for these tasks.
Limitations
- Does not surpass UMT-L or larger-data models on several tasks such as video QA and retrieval on some benchmarks.
- Video QA gains are limited; the authors note spatio-temporal masking mainly helps retrieval.
Experimental setup
Datasets: WebVid-2M CC3M Kinetics-710 MSRVTT DiDeMo ActivityNet Captions TGIF MSVD MSVD-QA ActivityNet-QA
Benchmarks: MSRVTT DiDeMo ActivityNet MSVD TGIF MSRVTT-QA MSVD-QA ActivityNet-QA
Metrics: Recall@1 accuracy CIDEr
Baselines: VIOLETv2 UMT Singularity VINDLU HiTeA LAVENDER Frozen ALPRO
Builds on: VIOLETv2 UMT masked autoencoders ActionCLIP
Closest papers by meaning
- Cluster-Wise Spatio-Temporal Masking for Efficient Video-Language Pretraining (CVPR 2026 · similarity 0.92)
- Understanding and Enhancing Mask-Based Pretraining towards Universal Representations (NeurIPS 2025 · similarity 0.88)
- Extending Video Masked Autoencoders to 128 frames (NeurIPS 2024 · similarity 0.88)
- Morphing Tokens Draw Strong Masked Image Models (ICLR 2025 · similarity 0.87)
- Video-Panda: Parameter-efficient Alignment for Encoder-free Video-Language Models (CVPR 2025 · similarity 0.87)
- Masked Diffusion Vision-Language Models for Temporal Action Localization (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.