Research map / Video / Video-language understanding
Robustness Analysis of Video-Language Models Against Visual and Language Perturbations
Madeline Chantry Schiappa, Shruti Vyas, Hamid Palangi, Yogesh S Rawat, Vibhav Vineet
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Perturb video rather than text | Video-language retrieval models · MSRVTT-P and YouCook2-P | relative robustness γ^r: MSRVTT-P: 0.91±0.15 vs 0.92±0.13; YouCook2-P: 0.55±0.37 vs 0.91±0.10 | Hurt |
| Use pre-training instead of training from scratch | Video-language retrieval models · MSRVTT-P and YouCook2-P | robustness and retrieval performance | Improved |
| Use FIT's ViT-based visual encoder rather than the studied CNN-based visual encoders | Video-language retrieval · MSRVTT-P visual noise | relative robustness γ^r: 0.81±0.21 vs 0.24±0.19 | Improved |
| Apply ChangeChar text perturbations | Video-language retrieval models · MSRVTT-P and YouCook2-P | relative robustness γ^r: MSRVTT-P: 0.71±0.11 to 0.81±0.09; YouCook2-P: 0.75±0.10 to 0.84±0.10 | Hurt |
| Perturb video and text together rather than separately | Video-language retrieval models · MSRVTT-P | relative robustness γ^r: 1 and 0.98 individually; 0.89 combined | Hurt |
| Fine-tune on the target dataset rather than evaluate zero-shot | Video-language retrieval models · MSRVTT-P | relative robustness γ^r: FIT video: 0.91±0.15 to 0.86±0.15; FIT text: 0.92±0.13 to 0.88±0.10 | Hurt |
| Fine-tune on the target dataset rather than evaluate zero-shot | Video-language retrieval models · YouCook2-P video perturbations | relative robustness γ^r: UniVL: 0.50±0.36 to 0.55±0.36 | Improved |
| Use cross-attention rather than a two-branch encoder | Video-language retrieval models · YouCook2-P | absolute robustness | Improved |
| Preserve alignment between video segments and their text rather than jumbling segments | Video-language retrieval models · YouCook2-P | robustness | Hurt |
| Perturb frame order within a text-aligned segment | Video-language retrieval models · MSRVTT-P and YouCook2-P | robustness | No clear effect |
| Train on the target dataset rather than rely on a larger pre-training dataset | Video-language retrieval models · MSRVTT-P gender-neutral text perturbations | retrieval performance | Hurt |
| Use Word2Vec rather than BERT text features | Zero-shot video-language retrieval · MSRVTT-P DropText, Positional, and ChangeChar | relative robustness γ^r: DropText: 0.57±0.32 vs 0.36±0.26; Positional: 0.78±0.15 vs 0.63±0.15; ChangeChar: 0.77±0.10 vs 0.71±0.10 | Mixed |
Closest papers by meaning
- Distilling Vision-Language Models on Millions of Videos (CVPR 2024 · similarity 0.89)
- ViLMA: A Zero-Shot Benchmark for Linguistic and Temporal Grounding in Video-Language Models (ICLR 2024 · similarity 0.88)
- Scaling Test-Time Robustness of Vision-Language Models via Self-Critical Inference Framework (CVPR 2026 · similarity 0.88)
- VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models (ICML 2026 · similarity 0.87)
- Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (CVPR 2025 · similarity 0.87)
- BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understanding (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
