Research map / Video / Video-language understanding
Efficient Transfer Learning for Video-language Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MSTA, a parameter-efficient multi-modal spatio-temporal adapter for ViCLIP with shared cross-modal projection and an LLM-description-guided consistency loss, evaluated across zero-shot, few-shot, base-to-novel, and fully-supervised video recognition.
Problem
Efficiently adapting video-language foundation models like ViCLIP to video action recognition while preserving generalizable knowledge and avoiding catastrophic forgetting and overfitting.
Approach
A multi-modal adapter with independent text/video down/up projections, a shared projection layer that receives gradients from both modalities, and spatial/temporal up-projections (3D conv) for video, plus a consistency loss that aligns the trainable text branch (template prompts) with the frozen branch on LLM-generated spatio-temporal descriptions.
Key results
- Base-to-novel: MSTA+LCC reaches HM 72.0 on K-400, 66.3 on HMDB-51, 82.9 on UCF-101, and 18.9 on SSv2, using ~8.7M tunable params.
- Few-shot (K=2) on HMDB-51, UCF-101, SSv2 with MSTA+LCC reaches 61.2, 86.1, 9.1 top-1, versus 60.4/88.0 for ViCLIP at K=4 range.
- Fully-supervised K-400: MSTA+LCC achieves 82.2 top-1 with 8.7M tunable parameters, versus 79.9 for ViCLIP and 82.3 for XCLIP (132M).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add MSTA and the description-guided consistency constraint | ViCLIP · Kinetics-400, HMDB-51, UCF-101, and Something-Something V2 base-to-novel | HM: 71.3 -> 72.0; 62.7 -> 66.3; 81.6 -> 82.9; 17.0 -> 18.9 | Helped |
| Add MSTA and the description-guided consistency constraint | ViCLIP · HMDB-51, UCF-101, and Kinetics-600 zero-shot | Top-1 accuracy: 51.9 ± 0.7 -> 55.8 ± 0.8; 77.1 ± 0.9 -> 78.7 ± 0.9; 71.5 ± 1.1 -> 74.5 ± 0.5 | Helped |
| Add MSTA and the description-guided consistency constraint | ViCLIP · Kinetics-400 fully supervised | Top-1 accuracy: +2.3 | Helped |
| Add adapters to both modalities rather than only the language branch (ablation) | MSTA · Base-to-novel generalization across four datasets | HM: 57.9 -> 60.1 | Helped |
| Add adapters to both modalities rather than only the vision branch (ablation) | MSTA · Base-to-novel generalization across four datasets | HM: 57.9 -> 60.1 | Helped |
| Add the shared projection layers (ablation) | MSTA · Base-to-novel generalization across four datasets | HM: 59.5 -> 60.1 | Helped |
| Increase shared-layer dimension from 64 to 256 (ablation) | MSTA · Base-to-novel generalization across four datasets | HM: 58.9 -> 60.1 | Helped |
| Increase shared-layer dimension from 256 to 512 (ablation) | MSTA · Base-to-novel generalization across four datasets | HM: 60.1 -> 60.0 | No clear effect |
| Set scaling factor λ to 0.05 instead of 0.005 (ablation) | MSTA · Base-to-novel generalization across four datasets | HM: 60.1 -> 50.4 | Hurt |
| Set consistency-constraint weight α to 1.0 instead of 0.1 (ablation) | MSTA · Base-to-novel generalization across four datasets | HM: 59.7 -> 60.1 | Helped |
| Increase consistency-constraint weight α from 1.0 to 5.0 (ablation) | MSTA · Base-to-novel generalization across four datasets | HM: 60.1 -> 59.8 | Hurt |
| Use 2 descriptions rather than 8 (ablation) | MSTA · Base-to-novel generalization across four datasets | HM: 59.7 -> 60.1 | Helped |
Surprising findings
- Using more LLM-generated descriptions (N=4 or 8) degrades performance, attributed to LLM hallucination producing noisy descriptions.
Limitations
- Relies on LLM-generated descriptions that can hallucinate, with performance degrading as description count grows
- Evaluated only on ViT-B/16 ViCLIP backbone
Experimental setup
Datasets: Kinetics-400 Kinetics-600 UCF-101 HMDB-51 SomethingSomething V2 InternVid-10M
Benchmarks: Kinetics-400 Kinetics-600 UCF-101 HMDB-51 SomethingSomething V2
Metrics: top-1 accuracy harmonic mean (HM)
Baselines: ViFi-CLIP OST MoTE XCLIP ActionCLIP Vita-CLIP A5 AdaptFormer LoRA ViCLIP
Builds on: ViCLIP AdaptFormer LoRA CLIP
Closest papers by meaning
- A Multimodal, Multi-Task Adapting Framework for Video Action Recognition (AAAI 2024 · similarity 0.89)
- OST: Refining Text Knowledge with Optimal Spatio-Temporal Descriptor for General Video Recognition (CVPR 2024 · similarity 0.89)
- Align Before Adapt: Leveraging Entity-to-Region Alignments for Generalizable Video Action Recognition (CVPR 2024 · similarity 0.89)
- MMA: Multi-Modal Adapter for Vision-Language Models (CVPR 2024 · similarity 0.89)
- AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportation (NeurIPS 2024 · similarity 0.88)
- READ-PVLA: Recurrent Adapter with Partial Video-Language Alignment for Parameter-Efficient Transfer Learning in Low-Resource Video-Language Modeling (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.