Research map / Video / Video-language understanding
VindLU: A Recipe for Effective Video-and-Language Pretraining
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add temporal attention across frames (ablation) | Image Transformer video-language baseline · MSR-VTT, DiDeMo, ActivityNet-Captions | average retrieval accuracy: +6.3% | Helped |
| Use late temporal attention instead of early per-layer temporal modeling (ablation) | Image Transformer video-language baseline · MSR-VTT, DiDeMo, ActivityNet-Captions | average retrieval accuracy | No clear effect |
| Fuse video features into text features (ablation) | Temporal Attention model · MSR-VTT, DiDeMo, ActivityNet-Captions | average retrieval accuracy: +3.6% | Helped |
| Fuse text features into video features (ablation) | Temporal Attention model · MSR-VTT, DiDeMo, ActivityNet-Captions | average retrieval accuracy: -1.3% | Hurt |
| Add bidirectional fusion on top of video-to-text fusion (ablation) | Temporal Attention model · MSR-VTT, DiDeMo, ActivityNet-Captions | average retrieval accuracy | No clear effect |
| Add masked language modeling (ablation) | VTC+VTM pretraining · MSR-VTT, DiDeMo, ActivityNet-Captions | average retrieval accuracy: +6.2% | Helped |
| Add masked video modeling (ablation) | VTC+VTM+MLM pretraining · MSR-VTT, DiDeMo, ActivityNet-Captions | average retrieval accuracy: +1% | Helped |
| Jointly pretrain on images and videos (ablation) | VindLU pretraining · MSR-VTT, DiDeMo, ActivityNet-Captions | average retrieval accuracy: +2.7% | Helped |
| Increase pretraining frames from 1 to 4 (ablation) | VindLU joint image-video pretraining · MSR-VTT, DiDeMo, ActivityNet-Captions | average retrieval accuracy: 1.7% | Helped |
| Increase pretraining frames beyond 4 (ablation) | VindLU joint image-video pretraining · MSR-VTT, DiDeMo, ActivityNet-Captions | average retrieval accuracy | No clear effect |
| Use multi-stage curriculum pretraining instead of single-stage 4-frame pretraining (ablation) | VindLU · MSR-VTT, DiDeMo, ActivityNet-Captions | average retrieval accuracy | No clear effect |
| Increase fine-tuning frames from 1 to 32 (ablation) | 4-frame pretrained VindLU · MSR-VTT, DiDeMo, ActivityNet-Captions | average retrieval accuracy: 5% | Helped |
Closest papers by meaning
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.89)
- ViLMA: A Zero-Shot Benchmark for Linguistic and Temporal Grounding in Video-Language Models (ICLR 2024 · similarity 0.89)
- SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understanding (ICLR 2025 · similarity 0.88)
- HourVideo: 1-Hour Video-Language Understanding (NeurIPS 2024 · similarity 0.88)
- Distilling Vision-Language Models on Millions of Videos (CVPR 2024 · similarity 0.87)
- LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.