Research map / Video / Video-language understanding
Can Text-to-Video Generation help Video-Language Alignment?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies whether synthetic videos from text-to-video generators, paired with LLM-generated negative captions, can improve video-language alignment, and proposes SYNVITA with alignment-based sample weighting and a semantic consistency loss.
Problem
Video-language alignment models trained with LLM-generated negative captions only learn some concepts as negatives, creating linguistic biases; real videos matching hard negatives are scarce, and synthetic videos from generators may be noisy and misaligned.
Approach
Generate videos from negative captions with text-to-video models and train a video LLM on them, weighting each synthetic video by how much better it aligns with its target caption than the real positive caption, plus a LCS-based triplet loss that focuses the model on semantic rather than appearance differences.
Key results
- SYNVITA (Video-LLaVA) improves over VideoCon (Video-LLaVA) on Human-Hard entailment by 1.12 AUC points and on SSv2-Events retrieval by 1.20 mAP.
- SYNVITA (mPLUG-Owl 7B) improves SSv2-Temporal mAP from 13.00 to 17.32 and ATP-Hard accuracy from 35.46 to 37.31 over VideoCon (mPLUG-Owl 7B).
- Preliminary study shows synthetic videos underperform on the in-distribution VideoCon (LLM) test set, with average AUC drops above 3 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SYNVITA synthetic-video training and semantic consistency regularization | VideoCon (mPLUG-Owl 7B) · VideoCon (LLM) | AUC-ROC: 88.39 -> 86.45 | Hurt |
| Add SYNVITA synthetic-video training and semantic consistency regularization | VideoCon (mPLUG-Owl 7B) · SSv2-Temporal | mAP: 13.00 -> 17.32 | Helped |
| Add SYNVITA synthetic-video training and semantic consistency regularization | VideoCon (mPLUG-Owl 7B) · SSv2-Events | mAP: 10.37 -> 12.54 | Helped |
| Add SYNVITA synthetic-video training and semantic consistency regularization | VideoCon (Video-LLaVA) · VideoCon (Human-Hard) | AUC-ROC: 75.74 -> 76.86 | Helped |
| Add alignment-based weighting and semantic consistency regularization (ablation) | SYNVITA (mPLUG-Owl 7B) · SSv2-Temporal, SSv2-Events, ATP-Hard | mAP / accuracy | Mixed |
| Use a fixed weight of 1.00 for all synthetic videos instead of alignment-based weighting (ablation) | SYNVITA (mPLUG-Owl 7B) · SSv2-Temporal | mAP: 12.54 vs 17.32 | Hurt |
| Weight videos only by alignment with the target caption, omitting comparison with the real caption (ablation) | SYNVITA (mPLUG-Owl 7B) · SSv2-Events | mAP: 9.40 vs 12.54 | Hurt |
| Use multiplicative weighting with inverse real-caption similarity (ablation) | SYNVITA (mPLUG-Owl 7B) · SSv2-Temporal | mAP: 15.52 vs 17.32 | Hurt |
| Use binary weighting based on whether target-caption alignment exceeds real-caption alignment (ablation) | SYNVITA (mPLUG-Owl 7B) · ATP-Hard | accuracy: 37.15 vs 37.31 | Hurt |
| Use all three generators' synthetic videos instead of CogVideoX videos (ablation) | SYNVITA (mPLUG-Owl 7B) · SSv2-Temporal | mAP: 17.32 vs 14.06 | Hurt |
| Add synthetic videos from CogVideoX (ablation) | VideoCon (mPLUG-Owl 7B) · SSv2-Temporal | mAP: 13.00 -> 17.32 | Helped |
Surprising findings
- Synthetic videos for FLIP and HALLUCINATION misalignments are often closer to the positive caption than the target negative, harming training.
- Naively training on videos from all three generators underperforms CogVideoX alone on downstream tasks, attributed to a high synthetic-to-real ratio and domain shift.
Limitations
- Synthetic videos harm performance on the in-distribution VideoCon (LLM) test set.
- Current open-source generators struggle with multi-event videos, making the consistency loss less effective on SSv2-Events.
- Only two hyperparameters but results depend on generator choice; no single generator is best across all tasks.
Experimental setup
Datasets: VideoCon MSR-VTT VATEX TEMPO SSv2-Temporal SSv2-Events ATP-Hard ActivityNet
Benchmarks: VideoCon (LLM) VideoCon (Human) VideoCon (Human-Hard) SSv2-Temporal SSv2-Events ATP-Hard
Metrics: AUC-ROC mAP accuracy
Baselines: VideoCon (mPLUG-Owl 7B) VideoCon (Video-LLaVA) VideoCLIP ImageBind (Video-Text) TACT VFC End-to-End VNLI mPLUG-Owl 7B Video-LLaVA
Builds on: VideoCon VQAScore mPLUG-Owl Video-LLaVA
Closest papers by meaning
- VideoCon: Robust Video-Language Alignment via Contrast Captions (CVPR 2024 · similarity 0.91)
- VideoComp: Advancing Fine-Grained Compositional and Temporal Alignment in Video-Text Models (CVPR 2025 · similarity 0.88)
- Distilling Vision-Language Models on Millions of Videos (CVPR 2024 · similarity 0.88)
- VidLA: Video-Language Alignment at Scale (CVPR 2024 · similarity 0.88)
- Neuro-Symbolic Evaluation of Text-to-Video Models using Formal Verification (CVPR 2025 · similarity 0.88)
- EvalCrafter: Benchmarking and Evaluating Large Video Generation Models (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.