Research map / Video / Video-language understanding
VideoComp: Advancing Fine-Grained Compositional and Temporal Alignment in Video-Text Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces VideoComp, a multi-event video-text compositionality benchmark (ActivityNet-Comp, YouCook2-Comp) with a hierarchical pairwise preference loss (CompLoss) and a short-clip concatenation pretraining strategy (CompPretrain) for a CLIP-style video model.
Problem
Video-text models lack fine-grained temporal and compositional alignment over continuous multi-event videos, and existing compositionality benchmarks focus on static images or single-event clips; dense multi-event video-text data is also scarce.
Approach
Build negative captions by reordering, action-word replacement, and segment-level mismatch on dense event captions, train a dual-encoder video CLIP with a hierarchical pairwise preference loss that ranks similarity by disruption level, and pretrain on concatenated short video-caption clips to simulate multi-event sequences.
Key results
- CompPretrain + CompLoss raises ActivityNet-Comp comprehensive accuracy from 18.9% (zero-shot VidCLIP-base) to 35.0%, with temporal reordering at 68.2%.
- Gemini-1.5-Pro reaches 43.9% comprehensive accuracy on ActivityNet-Comp zero-shot, with action-replace at 84.2%, above the fine-tuned VidCLIP-final on that disruption type.
- CompLoss alone outperforms NegCLIP on ActivityNet-Comp across all disruption types (comprehensive 31.2% vs 27.3%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CompLoss to VidCLIP-base fine-tuning (ablation) | VidCLIP-base · ActivityNet-Comp | Temp-reorder accuracy (%): 56.6 -> 65.4 | Helped |
| Add CompLoss to VidCLIP-base fine-tuning (ablation) | VidCLIP-base · ActivityNet-Comp | All-disruptions accuracy (%): 23.4 -> 31.2 | Helped |
| Pretrain using CompPretrain with stack size four (ablation) | VidCLIP-base with CompLoss · ActivityNet-Comp | All-disruptions accuracy (%): 31.2 -> 35.0 | Helped |
| Use CompPretrain with stack size four instead of stack size one (ablation) | VidCLIP-base (zero-shot) · ActivityNet-Comp | All-disruptions accuracy (%): 18.9 -> 21.6 | Helped |
| Replace standard contrastive loss with CompLoss (ablation) | VidCLIP-base fine-tuned on ActivityNet-Comp · ActivityNet-Comp | All-disruptions accuracy (%): 18.9 -> 31.2 | Helped |
| Use CompLoss instead of NegCLIP (ablation) | VidCLIP-base fine-tuned on ActivityNet-Comp · ActivityNet-Comp | All-disruptions accuracy (%): 27.3 -> 31.2 | Helped |
| Add CompPretrain to VidCLIP-base pretraining (ablation) | VidCLIP-base · ActivityNet-Comp | All-disruptions accuracy (%): 18.9 -> 21.6 | Helped |
| Add CompPretrain to VidCLIP-base pretraining (ablation) | VidCLIP-base · MSR-VTT | Recall@1: +1.8 / +1.0 | Helped |
| Add CompLoss and CompPretrain to VidCLIP-base (ablation) | VidCLIP-base · ActivityNet-Comp | Text-to-video Recall@1: 43.2 -> 42.8 | No clear effect |
| Add CompLoss and CompPretrain to VidCLIP-base (ablation) | VidCLIP-base · ActivityNet-Comp | Video-to-text Recall@1: 43.3 -> 43.1 | No clear effect |
Surprising findings
- Large LMMs such as Gemini-1.5-Pro perform well on action-word replacement but lag on temporal reordering and segment mismatch, where the compositionality-trained small model does better.
- Compositional training does not noticeably hurt retrieval accuracy (Recall@1 similar to standard finetuning).
Limitations
- Benchmark negatives are generated by an LLM (Gemini-1.5-Pro) and validated only by word-overlap heuristics, which may leave residual noise.
- Evaluation relies on binary classification and the comprehensive score is a multiplicative metric, so absolute scores are low and sensitive to the number of disruption types.
- Gap between the trained model and LMMs remains on action replacement; the approach is limited to a dual-encoder architecture.
Experimental setup
Datasets: ActivityNet-Captions YouCook2 VideoCC3M MSR-VTT ActivityNet
Benchmarks: ActivityNet-Comp YouCook2-Comp MSR-VTT ActivityNet
Metrics: binary classification accuracy comprehensive multiplicative accuracy Recall@1
Baselines: NegCLIP CLIP4Clip VideoCoCa VideoPrism InternVid Gemini-1.5-Flash Gemini-1.5-Pro Gemini-1.5-Flash-8B
Builds on: CLIP NegCLIP InfoNCE
Closest papers by meaning
- VideoCon: Robust Video-Language Alignment via Contrast Captions (CVPR 2024 · similarity 0.90)
- TOPA: Extending Large Language Models for Video Understanding via Text-Only Pre-Alignment (NeurIPS 2024 · similarity 0.90)
- VidLA: Video-Language Alignment at Scale (CVPR 2024 · similarity 0.89)
- T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation (CVPR 2025 · similarity 0.89)
- Can Text-to-Video Generation help Video-Language Alignment? (CVPR 2025 · similarity 0.88)
- VELOCITI: Benchmarking Video-Language Compositional Reasoning with Strict Entailment (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.