Research map / Vision-language and multimodal / CLIP and zero-shot
Align and Prompt: Video-and-Language Pre-Training With Entity Prompts
Check your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add VTC to MLM + VTM pre-training (ablation) | AlPro · MSRVTT retrieval | R1: 28.5 -> 32.8 | Helped |
| Add PEM to MLM + VTM pre-training (ablation) | AlPro · MSVD-QA | accuracy: 43.3 -> 46.3 | Helped |
| Use both PEM and VTC rather than VTC alone (ablation) | AlPro · MSVD-QA | accuracy: 45.5 -> 45.9 | No clear effect |
| Use both PEM and VTC rather than VTC alone (ablation) | AlPro · MSRVTT retrieval | R1: 32.8 -> 33.9 | Helped |
| Add prompt ensembling (ablation) | AlPro · MSRVTT retrieval (zero-shot) | R1: 22.6 -> 24.1 | Helped |
| Use 1000 entities instead of no PEM entities (ablation) | AlPro · MSRVTT retrieval (finetuning) | R1: 32.8 -> 33.9 | Helped |
| Increase PEM entity vocabulary from 1000 to 2000 (ablation) | AlPro · MSRVTT retrieval (zero-shot) | R1: 24.1 -> 22.4 | Hurt |
| Increase input frames from 2 to 8 (ablation) | AlPro · MSRVTT retrieval (finetuning) | R1: 25.7 -> 33.9 | Helped |
| Increase input frames from 8 to 16 (ablation) | AlPro · MSRVTT retrieval (finetuning) | R10: 73.2 -> 72.6 | Hurt |
| Use AlPro rather than FiT | video-text retrieval · MSRVTT retrieval (zero-shot) | R1: 18.7 -> 24.1 | Helped |
Closest papers by meaning
- TOPA: Extending Large Language Models for Video Understanding via Text-Only Pre-Alignment (NeurIPS 2024 · similarity 0.89)
- Stitching Segments and Sentences towards Generalization in Video-Text Pre-training (AAAI 2024 · similarity 0.88)
- Align Before Adapt: Leveraging Entity-to-Region Alignments for Generalizable Video Action Recognition (CVPR 2024 · similarity 0.88)
- Towards Balanced Alignment: Modal-Enhanced Semantic Modeling for Video Moment Retrieval (AAAI 2024 · similarity 0.88)
- LAMM: Label Alignment for Multi-Modal Prompt Learning (AAAI 2024 · similarity 0.87)
- VideoCon: Robust Video-Language Alignment via Contrast Captions (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.