AI Paper Atlas

Research map / Video / Video-language understanding

All in One: Exploring Unified Video-Language Pre-Training

Jinpeng Wang, Yixiao Ge, Rui Yan, Yuying Ge, Kevin Qinghong Lin, Satoshi Tsutsui, Xudong Lin, Guanyu Cai, Jianping Wu, Ying Shan, Xiaohu Qie, Mike Zheng Shou

CVPR 2023 · Video › Video-language understanding · approach: Video-language pretraining · 148 citations (OpenAlex)

Read the paper

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add temporal token rolling instead of flattening (ablation)TemporalSwap · LSMDC zero-shot multiple-choiceaccuracy: 51.40 -> 52.75Improved
add temporal token rolling instead of channel shift (ablation)TemporalSwap · MSR-VTT zero-shot multiple-choiceaccuracy: 73.16 -> 75.42Improved
increase the rolled-token ratio from 0 to 0.25 (ablation)TemporalSwap · LSMDC zero-shot multiple-choiceaccuracy: 47.21 -> 52.90Improved
increase the rolled-token ratio to 0.75 (ablation)TemporalSwap · LSMDC zero-shot multiple-choiceaccuracy: 52.90 -> 44.55Hurt
increase the rolled-token ratio to 0.75 (ablation)TemporalSwap · MSR-VTT zero-shot multiple-choiceaccuracy: 76.44 -> 69.45Hurt
add spatio-temporal position embeddings (ablation)TemporalSwap · LSMDC zero-shot multiple-choiceaccuracy: 53.2 -> 54.5Improved
add modality type embeddings alongside position embeddings (ablation)TemporalSwap · LSMDC zero-shot multiple-choiceaccuracy: 54.5 -> 55.7Improved
add temporal rolling from the first Transformer block rather than starting later (ablation)TemporalSwap · LSMDC zero-shot multiple-choiceaccuracy: 54.40 -> 56.15Improved
select a contiguous block of rolled tokens rather than random tokens (ablation)TemporalSwap · LSMDC zero-shot multiple-choiceaccuracy: 54.88 -> 55.81Improved
use VTM together with VTC rather than VTC alone (ablation)TemporalSwap retrieval fine-tuning · MSR-VTT text-to-video retrievalR@1: 36.7 -> 37.1Improved
add VTC to VTM fine-tuning (ablation)TemporalSwap retrieval fine-tuning · MSR-VTT text-to-video retrievalR@1: 33.3 -> 37.1Improved
use 3 frames for both pre-training and inference instead of one frame for pre-training and three for inference (ablation)TemporalSwap · LSMDC zero-shot multiple-choiceaccuracy: 35.16 -> 47.33Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy