Research map / Video / Video-language understanding
VideoCon: Robust Video-Language Alignment via Contrast Captions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds VideoCon, an LLM-generated dataset of semantically plausible contrast video captions with natural language explanations, and fine-tunes mPLUG-Owl-Video into Owl-Con for video-language entailment and explanation generation.
Problem
Video-language alignment models are not robust to semantically plausible contrastive changes in captions (entity, action, attribute, count, relation, hallucination, and event-order edits).
Approach
Generate type-controlled contrast captions and explanations with an LLM (PaLM-2) on temporally challenging video-caption pairs, then fine-tune a generative video-language model on entailment and NLE tasks so it judges alignment robustly and transfers zero-shot to retrieval and video QA.
Key results
- Owl-Con reaches 84.6 ROC-AUC on VideoCon (LLM) test and 78.3 on VideoCon (Human), versus 66.8 for Owl-Base on the human set.
- Owl-Con achieves 15.2 mAP on SSv2-Temporal and 11.4 mAP on SSv2-Events zero-shot text-to-video retrieval, the best among tested models.
- Owl-Con reaches 41.1% accuracy on zero-shot ATP-Hard video QA, above End-to-End VNLI at 39.0%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune mPLUG-Owl-Video on VideoCon contrast captions and explanations | Owl-Base · VideoCon (LLM) Test | ROC-AUC: 57.2 -> 84.6 | Helped |
| Fine-tune mPLUG-Owl-Video on VideoCon contrast captions and explanations | Owl-Base · VideoCon (Human) | ROC-AUC: 66.8 -> 78.3 | Helped |
| Fine-tune mPLUG-Owl-Video on VideoCon contrast captions and explanations | Owl-Base · VideoCon (Human-Hard) | ROC-AUC: 64.1 -> 74.4 | Helped |
| Fine-tune mPLUG-Owl-Video on VideoCon for NLE generation | Owl-Base · VideoCon (LLM) | Q2 entailment: 0.19 -> 0.50 | Helped |
| Fine-tune mPLUG-Owl-Video on VideoCon for NLE generation | Owl-Base · VideoCon (Human) | PaLM-2 entailment acc.(%): 39.6 -> 47.1 | Helped |
| Fine-tune mPLUG-Owl-Video on VideoCon | Owl-Base · SSv2-Temporal | mAP: 10.9 -> 15.2 | Helped |
| Fine-tune mPLUG-Owl-Video on VideoCon | Owl-Base · SSv2-Events | mAP: 6.8 -> 11.4 | Helped |
| Fine-tune mPLUG-Owl-Video on VideoCon | Owl-Base · ATP-Hard | Accuracy (%): 37.1 -> 41.1 | Helped |
| Use randomly selected captions as negatives instead of hard contrast captions (ablation) | Owl-Base fine-tuned on VideoCon · VideoCon (LLM) Test | ROC-AUC: 59.7 -> 84.6 | Hurt |
| Use randomly selected captions as negatives instead of hard contrast captions (ablation) | Owl-Base fine-tuned on VideoCon · SSv2-Events | mAP: 9.9 -> 11.4 | Hurt |
| Use randomly selected captions as negatives instead of hard contrast captions (ablation) | Owl-Base fine-tuned on VideoCon · ATP-Hard | Accuracy (%): 37.2 -> 41.1 | Hurt |
Surprising findings
- Random-negative fine-tuning (Owl-Rand) barely improves over the base model, showing that hard contrast negatives matter.
- The atemporal End-to-End VNLI model is competitive with or better than video-language models such as VideoCLIP and ImageBind on the downstream tasks.
Limitations
- Each contrast caption contains a single misalignment; multiple misalignments in one caption are not handled.
- Gains are smaller on out-of-distribution human-written contrast captions than on the LLM test set.
- Relies on automatically generated data from video captions that are human-written and well grounded; web-scale captions are not addressed.
Experimental setup
Datasets: MSR-VTT VaTeX TEMPO ActivityNet VideoCon (LLM) SSv2-Temporal SSv2-Events ATP-Hard
Benchmarks: VideoCon (LLM) test VideoCon (Human) VideoCon (Human-Hard) SSv2-Temporal SSv2-Events ATP-Hard
Metrics: ROC-AUC Q2 score entailment accuracy mAP accuracy
Baselines: VideoCLIP ImageBind Owl-Base Owl-Rand End-to-End VNLI TACT VFC CLIP
Builds on: mPLUG-Owl-Video VideoCLIP End-to-End VNLI ImageBind
Closest papers by meaning
- Can Text-to-Video Generation help Video-Language Alignment? (CVPR 2025 · similarity 0.91)
- Distilling Vision-Language Models on Millions of Videos (CVPR 2024 · similarity 0.90)
- VideoComp: Advancing Fine-Grained Compositional and Temporal Alignment in Video-Text Models (CVPR 2025 · similarity 0.90)
- VidLA: Video-Language Alignment at Scale (CVPR 2024 · similarity 0.90)
- Self-alignment of Large Video Language Models with Refined Regularized Preference Optimization (NeurIPS 2025 · similarity 0.89)
- LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.