Research map / Video / Video-language understanding
An Empirical Study of End-to-End Video-Language Transformers With Masked Visual Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SIF targets from Swin-B during video-text pre-training (ablation) | VIOLET · TGIF-Frame | accuracy: +0.7% | Helped |
| Add SIF targets from Swin-B during video-text pre-training (ablation) | VIOLET · DiDeMo retrieval | AveR: +5.8% | Helped |
| Use HOG targets rather than no MVM (ablation) | VIOLET · TGIF-Frame | accuracy: -0.8% | Hurt |
| Use optical-flow targets rather than no MVM (ablation) | VIOLET · DiDeMo retrieval | AveR: +1.1 | Helped |
| Combine SIF and pixel targets instead of SIF alone (ablation) | VIOLET · DiDeMo retrieval | AveR: 57.6 -> 55.1 | Hurt |
| Combine SIF and optical-flow targets instead of SIF alone (ablation) | VIOLET · DiDeMo retrieval | AveR: 57.6 -> 56.3 | Hurt |
| Use blockwise masking alongside random masking (ablation) | SIF MVM · DiDeMo retrieval | R1: 35.4 -> 36.4 | Helped |
| Increase random masking ratio from 15% to 30% (ablation) | SIF MVM · DiDeMo retrieval | R1: 35.4 -> 36.2 | Helped |
| Replace l1 with l2 loss for MVM (ablation) | SIF MVM · DiDeMo retrieval | AveR: 57.6 -> 55.0 | Hurt |
| Replace the 2-layer MLP head with one linear layer (ablation) | SIF MVM · DiDeMo retrieval | R1: 35.4 -> 31.3 | Hurt |
| Apply SIF MVM to image-text data rather than leaving image MVM off (ablation) | VIOLET · DiDeMo retrieval | AveR: -3.5% | Hurt |
| Apply MVM to video-text data only rather than both video-text and image-text (ablation) | VIOLET · DiDeMo retrieval | AveR: +0.3% | Helped |
Closest papers by meaning
- Distilling Vision-Language Models on Millions of Videos (CVPR 2024 · similarity 0.88)
- Frozen Transformers in Language Models Are Effective Visual Encoder Layers (ICLR 2024 · similarity 0.88)
- Video Language Model Pretraining with Spatio-temporal Masking (CVPR 2025 · similarity 0.88)
- PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding (NeurIPS 2025 · similarity 0.87)
- VILA: On Pre-training for Visual Language Models (CVPR 2024 · similarity 0.87)
- ViLMA: A Zero-Shot Benchmark for Linguistic and Temporal Grounding in Video-Language Models (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.