Research map / Video / Action recognition
Test-Time Zero-Shot Temporal Action Localization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes T3AL, a test-time adaptation method that adapts a frozen-encoder CoCa VLM's projectors per video for zero-shot temporal action localization without any training data.
Problem
Zero-shot temporal action localization (ZS-TAL) methods rely on training on large annotated video datasets, which may be unavailable and induce domain bias that hurts generalization to out-of-distribution videos.
Approach
Adapt a pre-trained VLM at test time on each unlabeled video: derive a video-level pseudo-label by averaging frame embeddings, refine frame scores with a BYOL-style self-supervised loss using confident frames as positives/negatives (updating only the vision and language projectors), and suppress proposals using text similarity of generated frame captions; parameters are reset after each video.
Key results
- On THUMOS14 (50%-50%) T3AL averages 10.4 mAP vs 3.5 for zero-shot CoCa, a gain over its own no-adaptation variant of +1.0 via test-time learning (T=0 gives 4.8 mAP).
- On ActivityNet-v1.3 (50%-50%) T3AL reaches 14.3 average mAP, versus 1.1 for CoCa and 13.3 for T3AL without adaptation.
- Training-based methods such as STALE (22.2 avg on THUMOS14 50%-50%) still outperform T3AL, which the authors attribute partly to oracle-based analysis showing gains up to 22.6 mAP when oracle information is used.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply test-time adaptation instead of stopping at T=0 | T3AL with CoCa ViT-L/14 · THUMOS14 50%-50% | average mAP: 4.8 -> 10.4 | Helped |
| Apply test-time adaptation instead of stopping at T=0 | T3AL with CoCa ViT-L/14 · THUMOS14 75%-25% | average mAP: 4.6 -> 9.2 | Helped |
| Apply test-time adaptation instead of stopping at T=0 | T3AL with CoCa ViT-L/14 · ActivityNet-v1.3 50%-50% | average mAP: 13.3 -> 14.3 | Helped |
| Apply test-time adaptation instead of stopping at T=0 | T3AL with CoCa ViT-L/14 · ActivityNet-v1.3 75%-25% | average mAP: 14.3 -> 15.4 | Helped |
| Add representation loss to BCE objective (ablation) | T3AL, vision projection only · THUMOS14 50%-50% | average mAP: 7.9 -> 10.3 | Helped |
| Add representation loss to separation loss (ablation) | T3AL, vision projection only · THUMOS14 50%-50% | average mAP: 8.4 -> 10.3 | Helped |
| Add language-projection adaptation and separation loss to representation loss (ablation) | T3AL · THUMOS14 50%-50% | average mAP: 10.3 -> 10.4 | Helped |
| Add text-guided region suppression (ablation) | T3AL · THUMOS14 50%-50% | average mAP: 10.1 -> 10.4 | Helped |
| Add text-guided region suppression (ablation) | T3AL · THUMOS14 75%-25% | average mAP: 8.8 -> 9.2 | Helped |
| Use a perfect video-level pseudo-label (ablation) | T3AL · THUMOS14 | average mAP: +1.2% mAP | Helped |
| Use the exact number of action regions (ablation) | T3AL · THUMOS14 | average mAP: +1.3% mAP | Helped |
| Select positive and negative refinement samples using ground-truth intervals (ablation) | T3AL · THUMOS14 | average mAP: 17.4% | Helped |
Surprising findings
- Training-based ZS-TAL methods (EffPrompt, STALE) drop by more than 15 mAP when evaluated cross-dataset on THUMOS14 compared with in-domain training.
- With all oracle components perfect, the test-time method reaches performance on par with in-domain trained state-of-the-art models.
Limitations
- Relies heavily on quality of positive and negative frame samples; random negatives may include titles or black screens.
- Video-level pseudo-label restricts each video to one action class, not handling concurrent actions.
- Still underperforms training-based methods such as STALE and EffPrompt on the reported benchmarks.
Experimental setup
Datasets: THUMOS14 ActivityNet-v1.3
Benchmarks: THUMOS14 ActivityNet-v1.3
Metrics: mAP at tIoU thresholds
Baselines: CLIP ViT-B/32 CLIP ViT-B/16 CoCa CLIP16 w/ Detector EffPrompt STALE UnLoc
Builds on: BYOL TENT TPT EffPrompt STALE
Closest papers by meaning
- Memory Matters: Boosting Training-Free Zero-Shot Temporal Action Localization with a Learnable Lookup Table (CVPR 2026 · similarity 0.95)
- Decompose and Conquer: Compositional Reasoning for Zero-Shot Temporal Action Localization (AAAI 2026 · similarity 0.90)
- Temporal Action Localization with Cross Layer Task Decoupling and Refinement (AAAI 2025 · similarity 0.90)
- Realigning Confidence with Temporal Saliency Information for Point-Level Weakly-Supervised Temporal Action Localization (CVPR 2024 · similarity 0.89)
- Bridge the Gap: From Weak to Full Supervision for Temporal Action Localization with PseudoFormer (CVPR 2025 · similarity 0.88)
- Text-Infused Attention and Foreground-Aware Modeling for Zero-Shot Temporal Action Detection (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.