Research map / Video / Action recognition
FROSTER: Frozen CLIP is A Strong Teacher for Open-Vocabulary Action Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FROSTER, which fine-tunes CLIP for video-specific action recognition while distilling from frozen CLIP through a residual feature distillation head to retain open-vocabulary generalization.
Problem
Open-vocabulary action recognition with CLIP: fine-tuning CLIP on video data improves seen classes but degrades generalization to unseen action categories.
Approach
Use the frozen CLIP as a feature-distillation teacher, but apply the distillation loss through a residual sub-network (identity plus zero-initialized two-layer MLP with scale alpha) so the tuned features can learn video-specific knowledge while staying close to the generalizable CLIP features; works with any architecture without weight-interpolation constraints.
Key results
- Under base-to-novel with ViT-B/16, FROSTER reaches 70.4 HM on K-400, 65.1 HM on HMDB-51, and 87.0 HM on UCF-101, outperforming Open-VCLIP (68.9, 58.7, 85.3).
- Under cross-dataset (trained on K-400), FROSTER achieves 84.8 on UCF, 54.8 on HMDB, and 74.8 on K-600, the best among compared methods.
- Applied to VCLIP, residual feature distillation raises K-600 accuracy from 65.9 to 74.8 and UCF* from 79.7 to 85.0 in the cross-dataset setting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add FROSTER residual feature distillation to VCLIP | VCLIP · K-600 cross-dataset | Top-1 accuracy: 65.9±1.0 -> 74.8$\pm$0.9 | Helped |
| Add residual feature distillation to vision encoder rather than no distillation (ablation) | VCLIP · UCF-101 cross-dataset | Top-1 accuracy: 79.7 -> 82.9 | Helped |
| Add residual feature distillation to text encoder rather than no distillation (ablation) | VCLIP · HMDB-51 cross-dataset | Top-1 accuracy: 48.5 -> 52.3 | Helped |
| Use an MLP projector for feature distillation instead of direct feature matching (ablation) | VCLIP vision encoder · UCF-101 cross-dataset | Top-1 accuracy: 81.8 -> 80.5 | Hurt |
| Use residual feature distillation instead of direct feature matching (ablation) | VCLIP vision encoder · UCF-101 cross-dataset | Top-1 accuracy: 81.8 -> 82.9 | Helped |
| Add GPT3.5 action-name descriptions (ablation) | FROSTER with vision and text distillation · K-600 cross-dataset | Top-1 accuracy: 73.0$\pm$0.7 -> 74.8$\pm$0.9 | Helped |
| Zero-initialize the second projector layer W2 (ablation) | FROSTER · K-600 cross-dataset | Top-1 accuracy: 74.4$\pm$1.0 -> 74.8$\pm$0.9 | Helped |
| Set α to 0.05 rather than 0.1 (ablation) | FROSTER · K-600 cross-dataset | Top-1 accuracy: 74.8$\pm$0.9 -> 74.3$\pm$0.9 | Hurt |
| Set β to 3 rather than 2 (ablation) | FROSTER · UCF-101 cross-dataset | Top-1 accuracy: 85.0 -> 85.0 | No clear effect |
Surprising findings
- Adding a plain two-layer MLP projector to the distillation target performs worse than direct distillation without a projector (e.g., K-600 67.9 vs 70.1 for the vision encoder), suggesting the projector weakens the generalizable supervision.
- On SSv2, FROSTER's gains on novel classes are modest (novel HM 14.6 vs 13.9 for ViFi-CLIP), which the authors attribute to the lack of cross-frame temporal information in CLIP pretraining.
Limitations
- Gains on the fine-grained temporal SSv2 dataset are modest, as CLIP lacks temporal pretraining.
- Relies on a hyperparameter alpha and beta that were tuned; performance depends on the balance between distillation and video-specific learning.
- Text augmentation depends on GPT-3.5-generated descriptions, adding dependency on an external proprietary model.
Experimental setup
Datasets: Kinetics-400 Kinetics-600 UCF-101 HMDB-51 Something-to-Something V2
Benchmarks: Kinetics-400 Kinetics-600 UCF-101 HMDB-51 Something-to-Something V2
Metrics: top-1 accuracy harmonic mean (HM)
Baselines: Frozen CLIP ActionCLIP X-CLIP VPT AIM ST-Adapter ViFi-CLIP Open-VCLIP Vita-CLIP Text4Vis ER-ZASR
Builds on: CLIP VCLIP Open-VCLIP ResNet residual learning
Closest papers by meaning
- Kronecker Mask and Interpretive Prompts are Language-Action Video Learners (ICLR 2025 · similarity 0.87)
- BDC-CLIP: Brownian Distance Covariance for Adapting CLIP to Action Recognition (ICML 2025 · similarity 0.87)
- Does Video-Text Pretraining Help Open-Vocabulary Online Action Detection? (NeurIPS 2024 · similarity 0.86)
- TACO: Towards Task-Consistent Open-Vocabulary Adaptation in Video Recognition (NeurIPS 2026 · similarity 0.85)
- Building a Multi-modal Spatiotemporal Expert for Zero-shot Action Recognition with CLIP (AAAI 2025 · similarity 0.85)
- ActionAtlas: A VideoQA Benchmark for Domain-specialized Action Recognition (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.