Research map / Video / Video-language understanding
MoTE: Reconciling Generalization with Specialization for Visual-Language to Video Knowledge Transfer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MoTE, a mixture of temporal experts with weight-merging regularization and test-time temporal feature modulation to transfer CLIP to video recognition while balancing zero-shot and close-set performance.
Problem
Adding temporal parameters to CLIP for video recognition improves close-set accuracy but degrades zero-shot generalization, creating a trade-off between the two settings.
Approach
Replace each temporal Transformer FFN with multiple experts trained from different initializations and routed stochastically (multinomial routing), merge expert weights for inference at constant cost, regularize the merged model over a weight-space region defined by temperature-softened merges (Weight Merging Regularization), and modulate temporal features at test time using text-proxy similarity
Key results
- With ViT-L/14, MoTE reaches 86.8% K400 top-1 with 4x3 views and 74.4 HM_ZS, achieving a trade-off score of 80.1, versus Open-VCLIP's 78.5 and Text4Vis's 75.3.
- With ViT-B/16 (8 frames), MoTE attains 83.4% on UCF-101 and 55.8% on HMDB-51 zero-shot, the best reported in that table.
- On few-shot HMDB-51, UCF-101 and SSv2 with K=2 to 16, MoTE outperforms MAXI and ViFi-CLIP on most settings, e.g., 61.3 vs 58.0 on HMDB-51 at K=2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace the single FFN with temporal experts (ablation) | Text4Vis with ViT-L/14 · UCF-101 zero-shot | accuracy: 82.6 -> 86.6 | Helped |
| replace the single FFN with temporal experts (ablation) | Text4Vis with ViT-L/14 · Kinetics-400 close-set | accuracy: 86.7 -> 85.5 | Hurt |
| add Weight Merging Regularization (ablation) | MoTE with ViT-L/14 · UCF-101 zero-shot | accuracy: 86.6 -> 87.2 | Helped |
| add MSE regularization (ablation) | MoTE with ViT-L/14 · HMDB-51 zero-shot | accuracy: 56.6 -> 57.0 | Helped |
| add temporal feature modulation (ablation) | MoTE with ViT-L/14 · HMDB-51 zero-shot | accuracy: 57.0 -> 61.4 | Helped |
| route training data to one expert rather than send all data to every expert (ablation) | MoTE with same expert initialization · UCF-101 zero-shot | accuracy: 82.6 -> 86.4 | Helped |
| use different expert initializations (ablation) | MoTE with all training data sent to experts · Kinetics-600 zero-shot | accuracy: 72.8 -> 79.0 | Helped |
| use 4 rather than 3 temporal experts per layer (ablation) | MoTE with ViT-L/14 · Kinetics-600 zero-shot | accuracy: 77.4 -> 78.9 | Helped |
| merge expert weights rather than ensemble logits (ablation) | MoTE with ViT-L/14 · UCF-101 zero-shot | accuracy: 86.3 -> 87.5 | Helped |
| use multinomial rather than random routing (ablation) | MoTE with ViT-L/14 · Kinetics-600 zero-shot | accuracy: 78.1 -> 78.9 | Helped |
| construct a region in weight space rather than use point merging regularization (ablation) | MoTE with ViT-L/14 · UCF-101 zero-shot | accuracy: 85.8 -> 87.5 | Helped |
| replace full-FFN experts with projection-level experts (ablation) | MoTE with ViT-L/14 · UCF-101 zero-shot | accuracy: 87.5 -> 84.2 | Hurt |
Surprising findings
- Random routing of experts performs worse than merging weights only through ensembling, and projection-level expert designs degrade performance, which the authors attribute to implicit knowledge sharing between experts.
- Continuous temperature sampling for the regularizer underperforms the discrete candidate set, and MSE-based weight merging regularization achieves notable results without labels.
Limitations
- Text space is limited to video category names, lacking richer semantic textual representations.
- On the harder Kinetics-600 zero-shot benchmark, MoTE underperforms some methods without additional parameters such as Open-VCLIP and MAXI.
Experimental setup
Datasets: Kinetics-400
Benchmarks: Kinetics-400 UCF-101 HMDB-51 Kinetics-600 Something-Something v2
Metrics: top-1 accuracy harmonic mean of zero-shot accuracy trade-off score (harmonic mean of K400 top-1 and HM_ZS)
Baselines: Text4Vis ViFi-CLIP Open-VCLIP FROSTER MAXI X-CLIP ActionCLIP ST-Adapter DiST OTI
Builds on: CLIP Text4Vis Mixture-of-Experts Open-VCLIP ViFi-CLIP
Closest papers by meaning
- Efficient Transfer Learning for Video-language Foundation Models (CVPR 2025 · similarity 0.88)
- OST: Refining Text Knowledge with Optimal Spatio-Temporal Descriptor for General Video Recognition (CVPR 2024 · similarity 0.87)
- AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportation (NeurIPS 2024 · similarity 0.87)
- Enhancing Temporal Understanding in Video-LLMs through Stacked Temporal Attention in Vision Encoders (NeurIPS 2025 · similarity 0.86)
- TACO: Towards Task-Consistent Open-Vocabulary Adaptation in Video Recognition (NeurIPS 2026 · similarity 0.86)
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.