Research map / Video / Video-language understanding
VELOCITI: Benchmarking Video-Language Compositional Reasoning with Strict Entailment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces VELOCITI, a short-video compositional reasoning benchmark built on VidSitu with seven agent/action/multi-event tests, and StrictVLE, a stricter yes/no entailment metric; evaluated Video-LLMs find them far below human accuracy.
Problem
Video-LLMs are assumed to handle compositional reasoning (associating agents, actions and events over time) in short videos, but this is not well measured; classic video-language entailment uses relative scoring that hides models' tendency to answer 'Yes' to both positive and negative captions.
Approach
Seven disentangled tests built from dense SRL annotations of VidSitu clips, with negatives created either by text-only LLM manipulation or from other entities in the same video, evaluated with StrictVLE which requires 'Yes' for the positive caption and 'No' for the negative caption independently.
Key results
- Best open model LLaVA-OneVision-72B reaches 43.2% average on StrictVLE and best closed model Gemini-1.5-Pro 49.3% on the subset, versus 93.0% human accuracy.
- Action understanding lags agent understanding for open models; OV-72B averages 35.8% on action tests vs 49.2% on agent tests.
- Under ClassicVLE, OV-72B reaches 83.6% average, showing relative scoring hides models that answer 'Yes' to both captions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase model size from LLaVA-OneVision-7B to LLaVA-OneVision-72B | LLaVA-OneVision · VELOCITI average (7 tests) | accuracy: 32.1 -> 43.1 | Helped |
| Use StrictVLE rather than ClassicVLE | Video-language entailment evaluation on VELOCITI · VELOCITI average (7 tests) | accuracy: 83.6 -> 43.1 | Hurt |
| Evaluate using multiple-choice instead of independent StrictVLE judgments | LLaVA-OneVision-72B · VELOCITI average (7 tests) | accuracy: 43.1 -> 76.0 | Helped |
| Replace text-inspired negation with in-video negation | GPT-4o · Agent Random and Action Adversarial versus Agent Binding and Action Binding | accuracy: 5.5% drop | Hurt |
| Test action understanding rather than agent understanding | LLaVA-OneVision-72B · VELOCITI | accuracy: 49.2% -> 35.8% | Hurt |
| Remove video input (ablation) | LLaVA-OneVision-72B · VELOCITI average (7 tests) | accuracy: 43.1 -> 8.1 | Hurt |
| Remove video input (ablation) | Qwen2-72B · VELOCITI average (7 tests) | accuracy: 43.1 -> 3.7 | Hurt |
| Replace the video with a single randomly selected frame (ablation) | LLaVA-OneVision-7B · VELOCITI average (7 tests) | accuracy: 32.1 -> 19.0 | Hurt |
| Use the single-image model with video inputs (ablation) | LLaVA-OneVision-7B · VELOCITI average (7 tests) | accuracy: 32.1 -> 19.1 | Hurt |
| Replace StrictVLE with ClassicVLE | LLaVA-OneVision-72B · VELOCITI average (7 tests) | accuracy: 43.1 -> 83.6 | Helped |
Surprising findings
- Multiple-choice evaluation shows strong choice bias, e.g. OV-7B picks option B 67.8% more than A, making MC unreliable.
- Smaller models answer 'Yes' to both positive and negative captions, while the text-only Qwen2-72B baseline and blind OV-72B score near random.
Limitations
- Closed and larger models (QVL-72B, closed models, humans) evaluated only on a 150-sample-per-test subset due to compute and cost constraints.
- Captions generated by LLaMA-3 from SRL annotations; videos limited to 10 s movie clips from VidSitu validation set (864 videos, 3101 pairs).
Experimental setup
Datasets: VidSitu
Benchmarks: VELOCITI
Metrics: accuracy StrictVLE accuracy ClassicVLE accuracy multiple-choice accuracy
Baselines: PLLaVA Video-LLaVA OwlCon Qwen2-VL LLaVA-OneVision Gemini-1.5-Flash Gemini-1.5-Pro GPT-4o VERA SigLIP ViFi-C NegCLIP
Builds on: VidSitu Video-Language Entailment (VideoCon) Winoground VERA
Closest papers by meaning
- STEP: Enhancing Video-LLMs' Compositional Reasoning by Spatio-Temporal Graph-guided Self-Training (CVPR 2025 · similarity 0.90)
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.89)
- A Benchmark for Omni-Modal Reasoning in Long Videos (NeurIPS 2026 · similarity 0.89)
- Black Swan: Abductive and Defeasible Video Reasoning in Unpredictable Events (CVPR 2025 · similarity 0.89)
- Look, Remember and Reason: Grounded Reasoning in Videos with Language Models (ICLR 2024 · similarity 0.89)
- FAVOR-Bench: A Comprehensive Benchmark for Fine-Grained Video Motion Understanding (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.