Research map / Video / Video-language understanding
ViLMA: A Zero-Shot Benchmark for Linguistic and Temporal Grounding in Video-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ViLMA, a zero-shot foiling benchmark with paired proficiency tests to probe temporal and linguistic grounding of video-language models, finding VidLMs barely beat image-language models.
Problem
Existing video-language evaluations rely on downstream tasks that fail to reveal fine-grained temporal visuo-linguistic capabilities, and pass/fail on main tests may reflect spurious heuristics rather than grounding.
Approach
A task-agnostic zero-shot counterfactual (foil) benchmark with five main tests (action counting, situation awareness, change of state, rare actions, spatial relations), each paired with a proficiency test; a combined P+T score credits a main-test success only if the paired proficiency test is also passed.
Key results
- BLIP2 achieves the highest average combined P+T score (57.5) and best Situation Awareness/Change of State results, outperforming most video-language models.
- Accounting for proficiency tests causes a striking drop in performance; e.g., CLIP falls from 50.9 on main test T to 46.2 on P+T in Change of State... and average P+T scores of video models are around 38-52.
- Video-language models perform close to image-language models and near random on Action Counting, Situation Awareness and Change of State, indicating limited temporal reasoning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| adding the proficiency-test requirement (combined P+T scoring) to main-test scoring | video-language and image-language models on ViLMA · ViLMA | P+T combined accuracy | Hurt |
| use image-language model BLIP2 instead of video-language models | situation awareness and overall ranking · ViLMA | P+T average | Helped |
| VideoLM temporal modelling (most VidLMs vs. image-LMs) | video-language models · ViLMA main tests | T pairwise accuracy | No clear effect |
Surprising findings
- Unimodal GPT-2 and OPT perform above random on Situation Awareness and Spatial Relations, attributed to plausibility biases in foil generation.
- Image-language models such as BLIP2 outperform most video-language models on several tests, including Situation Awareness.
Limitations
- Foil generation via MLMs and LMs can introduce plausibility biases that unimodal models exploit.
- Only about 61% of main-test samples passed human validation, and inter-annotator agreement is low to medium (alpha 0.1-0.4).
Experimental setup
Datasets: QUVA VidSitu RareAct Something-Something V2 YouCook2 COIN STAR HowTo100M
Benchmarks: ViLMA
Metrics: pairwise ranking accuracy combined P+T accuracy
Baselines: CLIP BLIP-2 GPT-2 OPT ClipBERT UniVL VideoCLIP FiT CLIP4Clip VIOLET X-CLIP MCQ
Builds on: VALSE QUVA VidSitu RareAct GRUEN
Closest papers by meaning
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.90)
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.90)
- PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding (NeurIPS 2025 · similarity 0.90)
- Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (CVPR 2025 · similarity 0.89)
- Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models (ICML 2024 · similarity 0.89)
- Distilling Vision-Language Models on Millions of Videos (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.