Research map / Vision-language and multimodal / Cross-modal learning
LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LanguageBind, which freezes an OpenCLIP-initialized language encoder and binds video, infrared, depth and audio encoders to it via contrastive learning (LoRA or full tuning), and releases the VIDAL-10M five-modal dataset construction (10M pairs).
Problem
Video-language pretraining frameworks do not extend easily to N>=3 modalities, and indirect alignment through images (ImageBind) may hurt language-related downstream performance; paired data for infrared, depth and audio with language is also scarce.
Approach
Use the frozen language encoder as the shared anchor: each non-language modality encoder (initialized from OpenCLIP ViT-L/14) is trained with contrastive loss directly against text, so all modalities align to language without images as intermediaries.
Key results
- Zero-shot text-to-video R@1 on MSR-VTT 42.8 (10M pairs, full tuning, ViT-L) vs InternVideo's 40.7; exceeds it by 1.9 points on MSR-VTT, 8.8 on MSVD, 6.3 on DiDeMo, 4.4 on ActivityNet per the paper.
- Zero-shot NYU-D depth top-1 accuracy 65.1 vs ImageBind 54.0; LLVIP infrared 87.2 vs ImageBind 63.4.
- Zero-shot ESC-50 audio accuracy 91.8 (LoRA) vs ImageBind 66.9, and 94.0 with full tuning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train CLIP4Clip with VIDAL-100k rather than HT100M-380k | CLIP4Clip · MSR-VTT | R@1: 32.0 -> 35.7 | Helped |
| Add audio embeddings to video-language retrieval | LanguageBind · MSR-VTT videos with audio | R@1: 41.4 -> 42.0 | Helped |
| Replace LoRA with full tuning (ablation) | LanguageBind · LLVIP | top-1 accuracy: 84.8 -> 85.1 | Helped |
| Use LoRA instead of full tuning (ablation) | LanguageBind · FLIR V1 and FLIR V2 | top-1 accuracy: 81.3 -> 81.6; 41.9 -> 46.6 | Mixed |
| Train from scratch rather than initialize from pretrained weights (ablation) | LanguageBind · LLVIP | top-1 accuracy: 85.1 -> 57.1 | Hurt |
| Increase training from 1 to 5 epochs (ablation) | LanguageBind · NYU-D, LLVIP, FLIR V1, and FLIR V2 | top-1 accuracy: 65.1 -> 64.5; 83.9 -> 81.1; 82.9 -> 85.0; 48.0 -> 44.7 | Mixed |
| Vary batch size from 1k to 512 or 2k (ablation) | LanguageBind · NYU-D, LLVIP, FLIR V1, and FLIR V2 | top-1 accuracy: NYU-D: 65.1, 63.9, 64.5; LLVIP: 83.9, 80.0, 78.6; FLIR V1: 82.9, 81.6, 85.2; FLIR V2: 48.0, 45.1, 47.9 | Mixed |
| Increase LoRA rank from 2 to 4 or 8 (ablation) | LanguageBind · NYU-D, LLVIP, FLIR V1, and FLIR V2 | top-1 accuracy: NYU-D: 65.1, 64.4, 64.7; LLVIP: 83.9, 78.0; FLIR V1: 82.9, 74.4; FLIR V2: 48.0, 45.8 | Hurt |
| Replace learnable temperature with fixed temperature (ablation) | LanguageBind · NYU-D, LLVIP, FLIR V1, and FLIR V2 | top-1 accuracy: NYU-D: 65.1, 63.0, 62.7; LLVIP: 83.9, 81.8, 83.1; FLIR V1: 82.9, 83.3, 80.3; FLIR V2: 48.0, 45.0, 43.2 | Mixed |
| Vary mask ratio from 0.5 to 0.0, 0.3, or 0.7 (ablation) | LanguageBind · NYU-D, LLVIP, FLIR V1, and FLIR V2 | top-1 accuracy: NYU-D: 65.1, 64.8, 62.7; LLVIP: 83.9, 80.3, 79.9, 81.5; FLIR V1: 82.9, 83.5, 84.2, 81.9; FLIR V2: 48.0, 43.2, 44.0, 42.5 | Mixed |
| Use different text sources instead of the ChatGPT-enhanced mPLUG caption (ablation) | LanguageBind · MSR-VTT | R@1: 33.5; 34.5; 35.8; 36.4 | Helped |
| Choose text source among raw, OFA, mPLUG, and ChatGPT-mPLUG captions (ablation) | LanguageBind · LLVIP, FLIR V1, FLIR V2, and NYU-D | top-1 accuracy: LLVIP: 83.9, 87.2, 84.6, 84.8; FLIR V1: 82.9, 80.6, 81.4, 81.6; FLIR V2: 48.0, 45.7, 46.8, 46.6; NYU-D: 61.5, 62.1, 63.9 | Mixed |
Surprising findings
- Training from scratch performs worst, but LoRA outperforms full tuning on several datasets (LLVIP, FLIR V1, Clotho) while using less time and memory.
- Larger batch size is not always better; 1024 was optimal in the ablation.
Limitations
- Infrared and depth data are synthetically generated by models (sRGB-TIR, GLPN) rather than captured with sensors, and the paper acknowledges model-bias limitations.
- VIDAL-10M release is only promised upon publication; the paper provides no compute hardware details.
- Several results (e.g., emergent retrieval) are on random 10% subsets and lack fair baselines.
Experimental setup
Datasets: VIDAL-10M YouTube-8M MSR-VTT COCO AVA HMDB-51 ImageNet HowTo100M LLVIP FLIR V1 FLIR V2 NYU-D
Benchmarks: MSR-VTT MSVD DiDeMo ActivityNet Kinetics-400 Kinetics-600 LLVIP FLIR V1 FLIR V2 NYU-D AudioSet ESC-50
Metrics: R@1 R@5 R@10 top-1 accuracy mAP
Baselines: ImageBind OpenCLIP InternVideo TVTSv2 UMT VideoCoCa OmniVL VALOR AVFIC CLIP4Clip
Builds on: CLIP OpenCLIP ImageBind LoRA MAE CLIP4Clip
Closest papers by meaning
- VidLA: Video-Language Alignment at Scale (CVPR 2024 · similarity 0.90)
- Self-alignment of Large Video Language Models with Refined Regularized Preference Optimization (NeurIPS 2025 · similarity 0.89)
- Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization (ICML 2024 · similarity 0.88)
- VideoCon: Robust Video-Language Alignment via Contrast Captions (CVPR 2024 · similarity 0.88)
- TOPA: Extending Large Language Models for Video Understanding via Text-Only Pre-Alignment (NeurIPS 2024 · similarity 0.88)
- LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videos (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.