Research map / Video / Video-language understanding
A Simple Recipe for Contrastively Pre-training Video-First Encoders Beyond 16 Frames
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Explores memory-efficient video-first ViT encoders trained with contrastive pre-training and video-to-text tuning, showing that high random tubelet masking (up to 75%) enables scaling to 256 frames (4.3 min at 1 FPS) and beats LLM-aggregation pipelines on YouCook2 and EgoSchema.
Problem
Video-language models built by transferring image-text encoders via late temporal fusion handle only short clips (~16 frames); video-first encoders with joint space-time attention are memory-bound, limiting how many frames can be processed for long-video understanding.
Approach
Systematically compare memory-efficient designs (factorized attention, adapters/LoRA, input tubelet masking, multi-resolution patchification) and find that high random input masking (75%) during contrastive pre-training is the most robust, enabling a two-stage short-to-long video encoder (LongViViT) that is frozen and fed to a frozen LM.
Key results
- LongViViT (~1B) with 256 frames reaches 20.3 Rouge-L zero-shot on YouCook2 vs 15.8 for ImageViT-to-Bard and 19.9 for PALI-to-Bard, and 56.8% on EgoSchema subset zero-shot vs 44.8% for PALI-to-Bard.
- Masking 75% of input tubelets during contrastive pre-training costs about 1% Recall@1 on zero-shot text-video retrieval and gives 2-3x memory savings.
- Joint space-time attention with 75% masking outperforms frame-level encodings on temporally rich benchmarks such as YouCook2 and VATEX, while factorized attention and adapters degrade more under masking.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use joint space-time attention instead of frame-level encodings | ViViT · YouCook2 | text-to-video Recall@1: 9.1 -> 12.3 | Helped |
| mask 75% of input video tokens during contrastive pre-training (ablation) | SHORTVIVIT-L · zero-shot video-text retrieval | Recall@1: an average of 1% absolute difference | Hurt |
| mask 75% of input video tokens during contrastive pre-training (ablation) | SHORTVIVIT-L · zero-shot video captioning | CIDEr | No clear effect |
| adapt SHORTVIVIT to longer contexts with LONGVIVIT training (ablation) | SHORTVIVIT · YouCook2 | Rouge-L: 1.8 Rouge-L points | Helped |
| use MLP Adapters to tune SHORTVIVIT for longer videos (ablation) | SHORTVIVIT · YouCook2 | Rouge-L | No clear effect |
| tune only the last four layers when adapting to longer videos (ablation) | LONGVIVIT · YouCook2 | Rouge-L | No clear effect |
| mask 30% of input video during video-to-text training and inference (ablation) | LONGVIVIT · video summarization | summarization performance | No clear effect |
| use LoRA instead of MLP Adapters for image-to-video adaptation (ablation) | ViViT · YouCook2 and VATEX | text-to-video Recall@1 | Hurt |
| use factorized attention instead of joint space-time attention (ablation) | ViViT · YouCook2 and VATEX | text-to-video Recall@1: 25% relative decrease | Hurt |
| use coarser temporal patchification (ablation) | video encoder · YouCook2 and VATEX | text-to-video Recall@1 | Hurt |
| remove video data from the training mixture (ablation) | video-to-text model · YouCook2 and VATEX | video-to-text Recall@1: up to 75% | Hurt |
| remove image data from the training mixture (ablation) | video-to-text model · MSR-VTT and ActivityNet | video-to-text Recall@1 | Hurt |
Surprising findings
- Blind Bard achieves SoTA on the full EgoSchema set (33.9%), and Bard alone is 27% on the released subset, showing some benchmarks are answerable without video.
- Simply masking up to 75% of the video during contrastive pre-training is more robust than architectural efficiency alternatives such as factorized attention, TubeViT, or parameter-efficient adapters.
Limitations
- Long-video gains are shown mainly on YouCook2 and EgoSchema; ActivityNet shows no benefit from longer training.
- Comparisons to VideoCoCa and Flamingo are not directly comparable due to different data, model sizes, and resolutions.
- Relies on proprietary Bard and PALI for captions and summaries, limiting reproducibility.
Experimental setup
Datasets: VTP (27M video-text pairs) HowTo100M VideoCC3M HowTo100M Summary MSR-VTT VATEX YouCook2 ActivityNet Captions EgoSchema Perceiver-style image datasets from Flamingo
Benchmarks: MSR-VTT VATEX YouCook2 ActivityNet Captions EgoSchema Perception Test
Metrics: Recall@1 CIDEr Rouge-L multiple-choice accuracy
Baselines: VideoCoCa Flamingo-3B SeViLA PALI-3 + Bard ImageViT-to-Bard SHORTVIVIT-to-Bard Blind Bard Factorized ST-ViViT TubeViT
Builds on: CoCa-style two-step recipe (Flamingo) ViViT TubeViT Perceiver Resampler VideoCoCa
Closest papers by meaning
- One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding (NeurIPS 2025 · similarity 0.89)
- Extending Video Masked Autoencoders to 128 frames (NeurIPS 2024 · similarity 0.88)
- Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization (ICML 2024 · similarity 0.88)
- FastVID: Dynamic Density Pruning for Fast Video Large Language Models (NeurIPS 2025 · similarity 0.87)
- Scaling Video-Language Models to 10K Frames via Hierarchical Differential Distillation (ICML 2025 · similarity 0.87)
- PAVE: Patching and Adapting Video Large Language Models (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.