Research map / Video / Video-language understanding
Verbs in Action: Improving Verb Understanding in Video-Language Models
Liliane Momeni, Mathilde Caron, Arsha Nagrani, Andrew Zisserman, Cordelia Schmid
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add LLM-generated verb-only hard negative captions with calibration (ablation) | CLIP video model on SMiT · MSR-VTT MC Verb_H | accuracy: 69.9 -> 78.0 | Improved |
| generate negatives with PaLM instead of random or antonym verbs (ablation) | hard negative generation · MSR-VTT MC Verb_H | accuracy: 73.6 (random) -> 78.0 (PaLM) | Improved |
| generate negatives with T5 instead of PaLM (ablation) | hard negative generation · MSR-VTT MC Verb_H | accuracy: 78.0 (PaLM) -> 75.1 (T5) | Mixed |
| remove calibration of hard negatives (ablation) | VFC hard negative loss · Kinetics-400 | top-1 accuracy: 55.8 -> 54.5 | Hurt |
| remove calibration of hard negatives (ablation) | VFC hard negative loss · MSR-VTT MC Verb_H | accuracy: 78.0 -> 80.5 | Improved |
| add verb phrase loss to hard negatives (ablation) | VFC · Kinetics-400 | top-1 accuracy: 55.8 -> 58.5 | Improved |
| add verb phrase loss to hard negatives (ablation) | VFC · MSR-VTT MC Verb_H | accuracy: 78.0 -> 76.3 | Hurt |
| increase max hard negative captions from 1 to 5 (ablation) | VFC · MSR-VTT MC Verb_H | accuracy: 76.3 -> 78.3 | Improved |
| use HardNeg-NCE instead of standard NCE (ablation) | VFC · MSR-VTT MC Verb_H | accuracy: 78.3 -> 80.5 | Improved |
| mine matching videos for generated captions instead of calibration (ablation) | VFC hard negatives · Kinetics-400 | top-1 accuracy: 55.8 -> 51.8 | Hurt |
| generate positive synonym-verb captions instead of negatives (ablation) | VFC · verb benchmarks | accuracy | Hurt |
| add input-output example pairs to PaLM prompt (ablation) | hard negative generation · MSR-VTT MC Verb_H | accuracy: 77.5 -> 78.0 | Improved |
Closest papers by meaning
- Kronecker Mask and Interpretive Prompts are Language-Action Video Learners (ICLR 2025 · similarity 0.87)
- VideoCon: Robust Video-Language Alignment via Contrast Captions (CVPR 2024 · similarity 0.87)
- E.T. Bench: Towards Open-Ended Event-Level Video-Language Understanding (NeurIPS 2024 · similarity 0.86)
- SlowFocus: Enhancing Fine-grained Temporal Understanding in Video LLM (NeurIPS 2024 · similarity 0.86)
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.86)
- Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoning (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
