Research map / Label-efficient and robust learning / OOD and anomaly detection

TACO: Towards Task-Consistent Open-Vocabulary Adaptation in Video Recognition

NeurIPS 2026 · Label-efficient and robust learning › OOD and anomaly detection · approach: Robust fine-tuning for OOD generalization · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TACO, which adapts CLIP for open-vocabulary video recognition using Relative Structure Distillation over random Gaussian hypersphere anchors plus a zero-initialized Specialization Projection, achieving SOTA on cross-dataset and base-to-novel benchmarks.

Problem

Fine-tuning CLIP on video data (e.g., Kinetics-400) restricts optimization to the in-distribution vocabulary, causing representation drift and cross-modal alignment shift in the OOD space, which degrades open-vocabulary generalization to unseen categories.

Approach

Distill the relative geometry (softmax similarity distributions) of visual and text embeddings against randomly sampled unit-hypersphere anchors from a frozen-like EMA teacher, so OOD structure is preserved without OOD data; and decouple the optimization space from the representation space via a residual linear specialization projection discarded at test time.

Key results

• Cross-dataset ViT-B/16: TACO reaches 85.6% UCF, 60.0% HMDB, 77.0% K600 validation splits, above FROSTER (84.8/54.8/74.8).
• Cross-dataset ViT-L/14: TACO reaches 91.4% UCF, 64.2% HMDB, 83.9% K600, above MoTE (88.7/61.4/78.4).
• Applied to ST-Adapter, Vita-CLIP, CLIP+LoRA and full fine-tuning, TACO improves zero-shot UCF/HMDB/K600 by 1.7 to 13.7 points.

Surprising findings

• Replacing the fine-tuned text encoder with the original CLIP text encoder barely changes zero-shot performance (66.3% vs 66.2%), suggesting OOD generalization is grounded in CLIP's original semantic space.
• Replacing the anchor term with a random D x D matrix (instead of hypersphere anchors) causes training collapse, and semantic anchors from LLM categories or WebVid text underperform Gaussian anchors.

Limitations

• Improvements are limited on temporally demanding datasets such as SSv2.
• Requires a relatively small EMA momentum whose optimal value may vary across datasets, limiting robustness.
• Gains on SSv2 are modest, as the method does not add temporal modules.

Experimental setup

Datasets: Kinetics-400 UCF-101 HMDB-51 Kinetics-600 Something-Something v2 WebVid-10M

Benchmarks: UCF-101 HMDB-51 Kinetics-600 Kinetics-400 Something-Something v2 ImageNet ImageNet-V2 ImageNet-S ImageNet-R ImageNet-A

Metrics: top-1 accuracy harmonic mean (HM)

Baselines: CLIP ActionCLIP X-CLIP VPT AIM ST-Adapter ViFi-CLIP Open-VCLIP FROSTER Open-MeDe MoTE OST

Builds on: CLIP Open-VCLIP FROSTER ViFi-CLIP knowledge distillation

Closest papers by meaning

• What Drives Test-Time Adaptation for CLIP? A Controlled Empirical Study from an Update Perspective (NeurIPS 2026 · similarity 0.87)
• A Multimodal, Multi-Task Adapting Framework for Video Action Recognition (AAAI 2024 · similarity 0.87)
• Align Before Adapt: Leveraging Entity-to-Region Alignments for Generalizable Video Action Recognition (CVPR 2024 · similarity 0.87)
• Test-Time Adaptation of Vision-Language Models for Open-Vocabulary Semantic Segmentation (NeurIPS 2025 · similarity 0.86)
• Efficient Transfer Learning for Video-language Foundation Models (CVPR 2025 · similarity 0.86)
• OST: Refining Text Knowledge with Optimal Spatio-Temporal Descriptor for General Video Recognition (CVPR 2024 · similarity 0.86)