Research map / Video / Video-language understanding
All in One: Exploring Unified Video-Language Pre-Training
Jinpeng Wang, Yixiao Ge, Rui Yan, Yuying Ge, Kevin Qinghong Lin, Satoshi Tsutsui, Xudong Lin, Guanyu Cai, Jianping Wu, Ying Shan, Xiaohu Qie, Mike Zheng Shou
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add temporal token rolling instead of flattening (ablation) | TemporalSwap · LSMDC zero-shot multiple-choice | accuracy: 51.40 -> 52.75 | Improved |
| add temporal token rolling instead of channel shift (ablation) | TemporalSwap · MSR-VTT zero-shot multiple-choice | accuracy: 73.16 -> 75.42 | Improved |
| increase the rolled-token ratio from 0 to 0.25 (ablation) | TemporalSwap · LSMDC zero-shot multiple-choice | accuracy: 47.21 -> 52.90 | Improved |
| increase the rolled-token ratio to 0.75 (ablation) | TemporalSwap · LSMDC zero-shot multiple-choice | accuracy: 52.90 -> 44.55 | Hurt |
| increase the rolled-token ratio to 0.75 (ablation) | TemporalSwap · MSR-VTT zero-shot multiple-choice | accuracy: 76.44 -> 69.45 | Hurt |
| add spatio-temporal position embeddings (ablation) | TemporalSwap · LSMDC zero-shot multiple-choice | accuracy: 53.2 -> 54.5 | Improved |
| add modality type embeddings alongside position embeddings (ablation) | TemporalSwap · LSMDC zero-shot multiple-choice | accuracy: 54.5 -> 55.7 | Improved |
| add temporal rolling from the first Transformer block rather than starting later (ablation) | TemporalSwap · LSMDC zero-shot multiple-choice | accuracy: 54.40 -> 56.15 | Improved |
| select a contiguous block of rolled tokens rather than random tokens (ablation) | TemporalSwap · LSMDC zero-shot multiple-choice | accuracy: 54.88 -> 55.81 | Improved |
| use VTM together with VTC rather than VTC alone (ablation) | TemporalSwap retrieval fine-tuning · MSR-VTT text-to-video retrieval | R@1: 36.7 -> 37.1 | Improved |
| add VTC to VTM fine-tuning (ablation) | TemporalSwap retrieval fine-tuning · MSR-VTT text-to-video retrieval | R@1: 33.3 -> 37.1 | Improved |
| use 3 frames for both pre-training and inference instead of one frame for pre-training and three for inference (ablation) | TemporalSwap · LSMDC zero-shot multiple-choice | accuracy: 35.16 -> 47.33 | Improved |
Closest papers by meaning
- Advancing High-Resolution Video-Language Representation With Large-Scale Video Transcriptions (CVPR 2022 · similarity 0.90)
- Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization (ICML 2024 · similarity 0.90)
- Long-Form Video-Language Pre-Training with Multimodal Temporal Contrastive Learning (NeurIPS 2022 · similarity 0.89)
- Enhancing Temporal Understanding in Video-LLMs through Stacked Temporal Attention in Vision Encoders (NeurIPS 2025 · similarity 0.88)
- LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment (ICLR 2024 · similarity 0.88)
- Dissecting Multimodality in VideoQA Transformer Models by Impairing Modality Fusion (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
