Research map / Vision-language and multimodal / Cross-modal learning
Clover: Towards a Unified Video-Language Alignment and Fusion Model
Jingjia Huang, Yinan Li, Jiashi Feng, Xinglong Wu, Xiaoshuai Sun, Rongrong Ji
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Clover pretraining with tri-modal alignment, semantic masking, and pair-wise ranking | Video-language retrieval model · DiDeMo fine-tune | R@10: 78.8 -> 85.6 | Improved |
| Use Clover pretraining with tri-modal alignment, semantic masking, and pair-wise ranking | Video-language retrieval model · MSRVTT fine-tune | R@10: 78.0 -> 79.4 | Improved |
| Use Clover pretraining with tri-modal alignment, semantic masking, and pair-wise ranking | Video-language retrieval model · LSMDC fine-tune | R@10: 45.9 -> 54.5 | Improved |
| Use Clover pretraining with tri-modal alignment, semantic masking, and pair-wise ranking | Video-language retrieval model · DiDeMo zero-shot | R@10: 63.6 -> 66.3 | Improved |
| Use Clover pretraining with tri-modal alignment, semantic masking, and pair-wise ranking | Video-language retrieval model · MSRVTT zero-shot | R@10: +0.3 | Improved |
| Use Clover pretraining with tri-modal alignment, semantic masking, and pair-wise ranking | Video-language retrieval model · LSMDC zero-shot | R@10: 32.2 -> 38.2 | Improved |
| Add tri-modal alignment (ablation) | Baseline pretraining · DiDeMo zero-shot ablation | R@1: 22.6 -> 24.7 | Improved |
| Add tri-modal alignment (ablation) | Baseline pretraining · MSRVTT zero-shot ablation | R@1: 19.8 -> 22.7 | Improved |
| Add semantic masking to TMA (ablation) | TMA pretraining · DiDeMo zero-shot ablation | R@1: 24.7 -> 25.3 | Improved |
| Add semantic masking to TMA (ablation) | TMA pretraining · MSRVTT zero-shot ablation | R@1: 22.7 -> 22.5 | Hurt |
| Add pair-wise ranking loss to TMA and semantic masking (ablation) | TMA plus semantic masking · DiDeMo zero-shot ablation | R@1: 25.3 -> 26.4 | Improved |
| Add pair-wise ranking loss to TMA and semantic masking (ablation) | TMA plus semantic masking · MSRVTT zero-shot ablation | R@1: 22.5 -> 23.4 | Improved |
Later papers that compared with this method
5 later accepted papers report CLOVER as a baseline in their results tables; 5 of them report a better number on most of the results they share with it.
- Disentangled Robot Learning via Separate Forward and Inverse Dynamics Pretraining (ICLR 2026 · better on 1 of 1 shared result)
- ReconVLA: Reconstructive Vision-Language-Action Model as Effective Robot Perceiver (AAAI 2026 · better on 1 of 1 shared result)
- WorldAgen: Unified State-Action Prediction with Test-Time World Model Training (AAAI 2026 · better on 1 of 1 shared result)
- Global and Local Semantic Completion Learning for Vision-Language Pre-Training (TPAMI 2025 · better on 3 of 3 shared results)
- Seeing What You Miss: Vision-Language Pre-Training With Semantic Completion Learning (CVPR 2023 · better on 3 of 3 shared results)
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Video-Panda: Parameter-efficient Alignment for Encoder-free Video-Language Models (CVPR 2025 · similarity 0.90)
- LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment (ICLR 2024 · similarity 0.89)
- MuLTI: Efficient Video-and-Language Understanding with Text-Guided MultiWay-Sampler and Multiple Choice Modeling (AAAI 2024 · similarity 0.88)
- Self-alignment of Large Video Language Models with Refined Regularized Preference Optimization (NeurIPS 2025 · similarity 0.88)
- TOPA: Extending Large Language Models for Video Understanding via Text-Only Pre-Alignment (NeurIPS 2024 · similarity 0.88)
- Video-Context Aligned Transformer for Video Question Answering (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
