Research map / Video / Video-language understanding
MV-Adapter: Multimodal Video Transfer Learning for Video Text Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MV-Adapter, a parameter-efficient adapter for CLIP-based video-text retrieval that adds temporal adaptation and cross-modality tying in bottleneck branches, matching or beating full fine-tuning with about 2.4% tunable parameters.
Problem
Full fine-tuning of CLIP for video-text retrieval requires storing a separate model per task, causing high storage costs; existing PETL adapters do not model temporal context and cross-modal interaction well for VTR.
Approach
Bottleneck adapters in both video and text branches, with a Temporal Adaptation module (lightweight transformer over frame tokens plus per-frame calibrated upsample weights) and Cross Modality Tying (modality-specific downsample weights generated from a shared cross-modal factor).
Key results
- On MSR-VTT with CLIP ViT-B/16, MV-Adapter reaches T2V R@Sum 202.1 and V2T R@Sum 205.9 vs 200.2 and 202.4 for full fine-tuning, using 2.39% tunable parameters.
- On LSMDC, MV-Adapter achieves T2V R@Sum 120.3 vs 112.9 for full fine-tuning with 2.42% tunable parameters.
- MV-Adapter uses about 60% of full fine-tuning GPU memory, and the paper states about 40% memory reduction in its efficiency analysis.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add temporal modeling with TRM to the Down&Up adapter (ablation) | MV-Adapter · MSR-VTT, T2V | R@Sum: 192.0 -> 197.5 | Helped |
| Add temporal modeling with TRM to the Down&Up adapter (ablation) | MV-Adapter · MSR-VTT, V2T | R@Sum: 195.3 -> 202.9 | Helped |
| Add temporal calibration after temporal modeling (ablation) | MV-Adapter with TRM · MSR-VTT, T2V | R@Sum: 197.5 -> 200.9 | Helped |
| Add temporal calibration after temporal modeling (ablation) | MV-Adapter with TRM · MSR-VTT, V2T | R@Sum: 202.9 -> 204.4 | Helped |
| Add Cross Modality Tying (CMT) (ablation) | MV-Adapter · MSR-VTT, T2V | R@Sum: 200.9 -> 202.1 | Helped |
| Add Cross Modality Tying (CMT) (ablation) | MV-Adapter · MSR-VTT, V2T | R@Sum: 204.4 -> 205.9 | Helped |
| Use both video and text adapter branches instead of only one branch (ablation) | MV-Adapter · MSR-VTT, T2V and V2T | R@Sum: -7.8/-5.3 and -9.2/-12.1 | Helped |
| Use MV-Adapter instead of full fine-tuning | CLIP ViT-B/16 · MSR-VTT, T2V | R@Sum: 200.2 -> 202.1 | Helped |
| Use MV-Adapter instead of full fine-tuning | CLIP ViT-B/16 · MSVD, T2V | R@Sum: 216.2 -> 214.8 | Hurt |
| Use MV-Adapter instead of full fine-tuning | CLIP ViT-B/16 · LSMDC, T2V | R@Sum: 112.9 -> 120.3 | Helped |
| Use MV-Adapter instead of full fine-tuning | CLIP ViT-B/16 · DiDemo, T2V | R@Sum: 199.6 -> 196.8 | Hurt |
| Use MV-Adapter instead of full fine-tuning | CLIP ViT-B/16 · ActivityNet, T2V | R@Sum: 201.5 -> 203.1 | Helped |
Surprising findings
- Naive adapters (AdaptFormer, Convpass, ST-Adapter) and frozen-backbone CLIP4Clip/Hunyuan fall well behind full fine-tuning when adapted to parameter-efficient VTR.
Limitations
- Fixed number of frames per video may miss fine movements in long videos.
- Audio-related captions are often mismatched since audio is not modeled.
- Only evaluated with CLIP dual-encoder backbones.
Experimental setup
Datasets: MSR-VTT MSVD LSMDC DiDemo ActivityNet
Benchmarks: MSR-VTT MSVD LSMDC DiDemo ActivityNet
Metrics: R@1 R@5 R@10 R@Sum
Baselines: Full fine-tuning AdaptFormer (Adaptpar/Adaptseq) Convpass ST-Adapter CM-Adapter CLIP4Clip Hunyuan
Builds on: AdaptFormer CLIP4Clip ST-Adapter Convpass
Closest papers by meaning
- A Multimodal, Multi-Task Adapting Framework for Video Action Recognition (AAAI 2024 · similarity 0.87)
- MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (CVPR 2024 · similarity 0.87)
- Towards Balanced Alignment: Modal-Enhanced Semantic Modeling for Video Moment Retrieval (AAAI 2024 · similarity 0.86)
- Efficient Transfer Learning for Video-language Foundation Models (CVPR 2025 · similarity 0.86)
- READ-PVLA: Recurrent Adapter with Partial Video-Language Alignment for Parameter-Efficient Transfer Learning in Low-Resource Video-Language Modeling (AAAI 2024 · similarity 0.86)
- TempMe: Video Temporal Token Merging for Efficient Text-Video Retrieval (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.