Research map / Vision-language and multimodal / CLIP and zero-shot
ViLT-CLIP: Video and Language Tuning CLIP with Multimodal Prompt Learning and Scenario-Guided Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts frozen CLIP ViT-B/16 to video recognition and retrieval via multimodal (vision+text) deep prompts with a cross-modal linear mapping and a cosine-similarity constraint to hand-crafted prompt embeddings.
Problem
Adapting image-based CLIP to video tasks while balancing fully supervised accuracy against zero-shot, few-shot, and base-to-novel generalization; existing fine-tuning hurts generalization and single-branch prompting is suboptimal.
Approach
Learn branch-aware hierarchical deep prompts in both the vision and text encoders of a frozen CLIP, link them with a learnable cross-modal linear mapping, and add a scenario-guided loss that keeps learnable text embeddings close to the hand-crafted "a video of a person doing [CLS]" embeddings.
Key results
- Fully supervised on Kinetics-400 with ViT-B/16 (16 frames): 77.6% top-1, 0.7% above A5 with a frozen vision encoder.
- Base-to-novel: harmonic mean gains of +3.8 on HMDB-51 and +2.7 on UCF-101 over prior best methods.
- Text-video retrieval on MSR-VTT: R@1 44.8 vs. 43.1 for CLIP4Clip-meanP and 36.7 for A5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add scenario-guided optimization to text and vision prompts (ablation) | ViLT-CLIP · Kinetics-400 | Top-1 accuracy: 76.46 -> 77.63 | Helped |
| Add multimodal prompt linear layer to text and vision prompts (ablation) | ViLT-CLIP · Kinetics-400 | Top-1 accuracy: 75.83 -> 76.46 | Helped |
| Add vision prompts to text prompts (ablation) | ViLT-CLIP · Kinetics-400 | Top-1 accuracy: 72.97 -> 75.83 | Helped |
| Add text prompts (ablation) | CLIP B/16 · Kinetics-400 | Top-1 accuracy: 40.10 -> 72.97 | Helped |
| Use multimodal rather than independent prompting (ablation) | Video-text retrieval on MSR-VTT (9K) · MSR-VTT (9K) | R@1: 43.5 -> 44.8 | Helped |
| Use 4+X text prompts and X vision prompts instead of 4+X prompts in both branches (ablation) | Independent V-L prompting · MSR-VTT (9K) | R@1: 43.3 -> 39.1 | Hurt |
| Increase vision prompts from 6+X to 8+X with 4+X text prompts (ablation) | Independent V-L prompting · MSR-VTT (9K) | R@1: 42.6 -> 43.6 | Helped |
| Increase vision prompts from 6+X to 10+X with 6+X text prompts (ablation) | Independent V-L prompting · MSR-VTT (9K) | R@1: 43.5 -> 44.3 | Helped |
| Apply ViLT-CLIP instead of ViFi-CLIP | CLIP video recognition · HMDB-51, 16-shot | accuracy: +2.8 | Helped |
| Apply ViLT-CLIP instead of ViFi-CLIP | CLIP video recognition · UCF-101, 16-shot | accuracy: +0.8 | Helped |
| Apply ViLT-CLIP instead of ViFi-CLIP | CLIP video recognition · SSv2, 16-shot | accuracy: +0.8 | Helped |
| Apply ViLT-CLIP instead of the previous best method | CLIP video recognition · UCF-101, 4-shot | accuracy: -0.8 | Mixed |
Surprising findings
- Fully fine-tuning CLIP for strong supervised performance reduces zero-shot, few-shot, and base-to-novel generalization, while freezing preserves generalization at a cost to supervised accuracy.
Limitations
- Mixed comparisons on retrieval because pre-training datasets differ, so fair comparison is not possible per the authors.
- Reported zero-shot results use a single setting trained on Kinetics-400 and evaluated on HMDB-51 and UCF-101 only.
Experimental setup
Datasets: Kinetics-400 HMDB-51 UCF-101 Something-Something v2 MSR-VTT MSVD LSMDC DiDeMo
Benchmarks: Kinetics-400 HMDB-51 UCF-101 Something-Something v2 MSR-VTT MSVD LSMDC DiDeMo
Metrics: top-1 accuracy top-5 accuracy harmonic mean R@1 R@5 MnR
Baselines: ActionCLIP X-CLIP A5 ViFi-CLIP CLIP4Clip VL prompting Vanilla CLIP
Builds on: CLIP CoOp MaPLe VPT ViFi-CLIP
Closest papers by meaning
- Learning to Learn Better Visual Prompts (AAAI 2024 · similarity 0.90)
- Learning to Prompt with Text Only Supervision for Vision-Language Models (AAAI 2025 · similarity 0.90)
- IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.90)
- Aggregate-and-Adapt Natural Language Prompts for Downstream Generalization of CLIP (NeurIPS 2024 · similarity 0.90)
- PromptKD: Unsupervised Prompt Distillation for Vision-Language Models (CVPR 2024 · similarity 0.90)
- A Hard-to-Beat Baseline for Training-free CLIP-based Adaptation (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.