Research map / Vision-language and multimodal / CLIP and zero-shot
A Multimodal, Multi-Task Adapting Framework for Video Action Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes M²-CLIP, a multimodal, multi-task CLIP adapting framework for video action recognition that combines a visual TED-Adapter (temporal enhancement and difference), text adapters, and a four-head multi-task decoder to keep supervised accuracy while preserving zero-shot transfer.
Problem
Parameter-efficient transfer of CLIP to video action recognition often uses only the visual branch, which sacrifices zero-shot generalization, while naively freezing both CLIP branches lowers supervised accuracy.
Approach
Insert trainable adapters in both visual and text branches of frozen CLIP: a TED-Adapter that combines global 1D temporal convolution with local frame-difference 2D convolution, plus a text adapter; train with a multi-task decoder (contrastive, cross-modal classification, cross-modal masked language modeling, and visual classification heads).
Key results
- M²-CLIP-B/16 with 8 frames reaches 84.1% top-1 on K400 with 32 frames (83.4% at 8 frames) using about 16M tunable parameters.
- Zero-shot UCF101 accuracy of 78.7% vs 75.0% for Vita-CLIP-B/16 and 72.0% for X-CLIP-B/16.
- On SSv2 with 32 frames, M²-CLIP reaches 69.1% top-1, and surpasses Vita-CLIP by over 18%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add TED-Adapter (ablation) | CLIP · K400 | Top-1 accuracy: 56.5 -> 80.5 | Helped |
| add Text-Adapter to the TED-Adapter (ablation) | CLIP with TED-Adapter · K400 | Top-1 accuracy: 80.5 -> 81.4 | Helped |
| add Multi-Task Decoder (ablation) | CLIP with multimodal adapters · K400 | Top-1 accuracy: 81.4 -> 83.4 | Helped |
| add cross-modal classification (CMC) to contrastive learning (ablation) | M²-CLIP multi-task decoder · K400 | Top-1 accuracy: +0.7% | Helped |
| add cross-modal masked language modeling (CMLM) (ablation) | M²-CLIP multi-task decoder · K400 | Top-1 accuracy | Helped |
| add visual classification (VC) head (ablation) | M²-CLIP multi-task decoder · K400 | Top-1 accuracy: 83.4% | Helped |
| use temporal enhancement (TE) rather than temporal difference (TD) alone (ablation) | Video TED-Adapter · K400 | Top-1 accuracy | Helped |
| combine temporal enhancement (TE) and temporal difference (TD) (ablation) | Video TED-Adapter · K400 | Top-1 accuracy | Helped |
| place TED-Adapters in deeper rather than shallow layers (ablation) | Video TED-Adapter · K400 | Top-1 accuracy | Helped |
| add TED-Adapter components in parallel rather than sequentially (ablation) | Video TED-Adapter · K400 | Top-1 accuracy | Helped |
| increase the number of Text-Adapters beyond one (ablation) | M²-CLIP · K400 | Top-1 accuracy | Mixed |
| use M²-CLIP instead of Vita-CLIP-B/16 | CLIP-based video action recognition · UCF101 zero-shot | accuracy: 75.0 ± 0.6 -> 78.7 ± 1.2 | Helped |
Surprising findings
- Adding only a single text adapter performs best; more text adapters reduce performance and zero-shot generalization, attributed to overfitting on label-only text.
Limitations
- Supervised results trail larger ViT-L methods such as BIKE and ILA, which use about 14x more tunable parameters.
- Multiple text adapters overfit to label-only text, limiting the design to a single text adapter.
Experimental setup
Datasets: Kinetics-400 Something-Something-V2
Benchmarks: Kinetics-400 Something-Something-V2 UCF101 HMDB51
Metrics: Top-1 accuracy Top-5 accuracy zero-shot accuracy tunable parameters GFLOPs
Baselines: ST-Adapter AIM EVL DUALPATH Vita-CLIP STAN ActionCLIP X-CLIP BIKE ILA Uniformer V2 S-ViT
Builds on: CLIP ST-Adapter ActionCLIP Vita-CLIP AIM
Closest papers by meaning
- Efficient Transfer Learning for Video-language Foundation Models (CVPR 2025 · similarity 0.89)
- MV-Adapter: Multimodal Video Transfer Learning for Video Text Retrieval (CVPR 2024 · similarity 0.87)
- Weakly Supervised Temporal Action Localization via Dual-Prior Collaborative Learning Guided by Multimodal Large Language Models (CVPR 2025 · similarity 0.87)
- Kronecker Mask and Interpretive Prompts are Language-Action Video Learners (ICLR 2025 · similarity 0.87)
- Modality-Collaborative Test-Time Adaptation for Action Recognition (CVPR 2024 · similarity 0.87)
- TACO: Towards Task-Consistent Open-Vocabulary Adaptation in Video Recognition (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.