Research map / Vision-language and multimodal / CLIP and zero-shot
DiscoVLA: Discrepancy Reduction in Vision, Language, and Alignment for Parameter-Efficient Video-Text Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiscoVLA, a parameter-efficient CLIP adaptation for video-text retrieval that fuses image- and video-level features (IVFusion), learns pseudo-image-caption alignment (PImgAlign), and distills image-level alignment into video-level alignment (AlignDistill).
Problem
Transferring image-level CLIP to video-text retrieval in a parameter-efficient way suffers from vision, language, and alignment discrepancies between image-level and video-level vision-language matching, which prior PETL methods only partially address.
Approach
Combine LoRA-based image-level attention with a lightweight adapter-fused CLS-token spatio-temporal branch (IVFusion) for vision and text encoders, add training-only pseudo image captions from LLaVA-NeXT with fine-grained max-similarity image-level alignment, and use KL distillation from image-level to video-level similarity distributions.
Key results
- On MSRVTT with CLIP ViT-B/16, DiscoVLA reaches 50.5% text-to-video R@1, 2.2% above prior methods, and 7.5% higher R@sum.
- With CLIP ViT-B/32 on MSRVTT, DiscoVLA achieves 47.0% t2v R@1 and 47.7% v2t R@1 using 0.56M trainable parameters.
- On LSMDC with ViT-B/32, DiscoVLA reaches 23.6% t2v R@1 versus 21.4% for DGL, with 0.56M trainable parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add IVFusion to LoRA (ablation) | LoRA · MSRVTT | R@sum: 193.0 -> 201.7 | Helped |
| Add PImgAlign to IVFusion (ablation) | IVFusion · MSRVTT | R@sum: 201.7 -> 201.4 | No clear effect |
| Add AlignDistill to IVFusion (ablation) | IVFusion · MSRVTT | R@sum: 201.7 -> 199.2 | No clear effect |
| Combine PImgAlign and AlignDistill (ablation) | IVFusion · MSRVTT | R@sum: 201.7 -> 202.8 | Helped |
| Replace Image-Level Attn with Video-Level Attn (ablation) | LoRA attention · MSRVTT | R@sum: 193.0 -> 197.4 | Helped |
| Remove the adapter from IVFusion Attn (ablation) | IVFusion Attn · MSRVTT | R@sum: 201.7 -> 197.4 | Hurt |
| Use video-level similarity instead of fine-grained image-level similarity in PImgAlign (ablation) | PImgAlign · MSRVTT | R@sum: 202.8 -> 199.8 | Hurt |
| Use matched image-caption pairs for PImgAlign similarity (ablation) | PImgAlign · MSRVTT | R@sum: 199.8 -> 201.7 | Helped |
| Use fine-grained image-level similarity across all image-caption pairs (ablation) | PImgAlign · MSRVTT | R@sum: 201.7 -> 202.8 | Helped |
| Generate pseudo image captions without video caption guidance (ablation) | PImgAlign · MSRVTT | R@sum: 202.8 -> 201.2 | Hurt |
Surprising findings
- Image-level PImgAlign or AlignDistill alone yield only limited gains, but combined they complement each other to improve video-level alignment.
Limitations
- PImgAlign requires an MLLM (LLaVA-NeXT) to generate pseudo image captions during training, adding computational overhead.
- Pseudo image captions cannot be generated at inference because ground-truth video captions are unavailable, so PImgAlign is excluded at test time.
Experimental setup
Datasets: MSRVTT LSMDC ActivityNet DiDeMo
Benchmarks: MSRVTT LSMDC ActivityNet DiDeMo
Metrics: R@1 R@5 R@10 R@sum MnR
Baselines: CLIP4Clip-meanP Prompt Adapter LoRA RAP VoP DGL MV-Adapter TempMe
Builds on: CLIP CLIP4Clip LoRA VoP DGL
Closest papers by meaning
- TOPA: Extending Large Language Models for Video Understanding via Text-Only Pre-Alignment (NeurIPS 2024 · similarity 0.89)
- Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehension (NeurIPS 2025 · similarity 0.88)
- Distilling Vision-Language Models on Millions of Videos (CVPR 2024 · similarity 0.88)
- Towards Balanced Alignment: Modal-Enhanced Semantic Modeling for Video Moment Retrieval (AAAI 2024 · similarity 0.88)
- Text as Partial Constraint: Core–Residual Alignment for Robust Vision–Language Learning (NeurIPS 2026 · similarity 0.88)
- VidLA: Video-Language Alignment at Scale (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.