Research map / Vision-language and multimodal / CLIP and zero-shot
Text-Infused Attention and Foreground-Aware Modeling for Zero-Shot Temporal Action Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a cross-modal ZSTAD baseline with mutual cross-attention and Ti-FAD, which adds text-infused cross attention (salient attentive mask) and a foreground-aware head to counter common-action bias on THUMOS14 and ActivityNet v1.3.
Problem
Zero-shot temporal action detection (ZSTAD) for unseen action categories in untrimmed videos; existing foreground-based methods fuse text and visual features only after pre-extracted proposals, and cross-modal models show a common-action bias that over-focuses on sub-actions shared with seen classes
Approach
Integrate text and visual features throughout the detection pipeline with cross-attention, then use a class-score-derived Gaussian Salient Attentive Mask as an attention bias in cross-attention (TiCA) and a soft/hard foreground-aware head to suppress background.
Key results
- Ti-FAD with I3D features reaches 41.2% average mAP on THUMOS14 under 50%-50% split (+11.0 over prior SOTA), and 46.8% under 75%-25%.
- Ti-FAD achieves 32.0% average mAP on ActivityNet v1.3 50%-50% (+5.4 over prior SOTA).
- Ablation shows SAM alone raises THUMOS14 50%-50% average mAP from 34.5 to 35.3, and adding H-FAD and S-FAD further improves to 41.2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the cross-modal baseline with full Ti-FAD | ZSTAD with I3D + CLIP-B · THUMOS14 50%-50% unseen | mAP@Avg: 34.5 -> 41.2 | Helped |
| Replace the cross-modal baseline with full Ti-FAD | ZSTAD with I3D + CLIP-B · THUMOS14 75%-25% unseen | mAP@Avg: 37.4 -> 46.8 | Helped |
| Replace the cross-modal baseline with full Ti-FAD | ZSTAD with I3D + CLIP-B · ActivityNet v1.3 50%-50% unseen | mAP@Avg: 26.6 -> 32.0 | Helped |
| Replace the cross-modal baseline with full Ti-FAD | ZSTAD with I3D + CLIP-B · ActivityNet v1.3 75%-25% unseen | mAP@Avg: 28.3 -> 34.7 | Helped |
| Add SAM to the baseline (ablation) | Ti-FAD · THUMOS14 50%-50% unseen | mAP@Avg: 34.5 -> 35.3 | Helped |
| Add SAM to the baseline (ablation) | Ti-FAD · THUMOS14 75%-25% unseen | mAP@Avg: 37.4 -> 40.0 | Helped |
| Add S-FAD to the baseline (ablation) | Ti-FAD · THUMOS14 50%-50% unseen | mAP@Avg: 34.5 -> 36.1 | Helped |
| Add H-FAD to the baseline (ablation) | Ti-FAD · THUMOS14 50%-50% unseen | mAP@Avg: 34.5 -> 38.4 | Helped |
| Add SAM + S-FAD + H-FAD (ablation) | Ti-FAD · THUMOS14 75%-25% unseen | mAP@Avg: 37.4 -> 46.8 | Helped |
| Use text-to-vision instead of vision-to-text cross-attention (ablation) | TiCA · THUMOS14 50%-50% unseen | mAP@Avg: 41.2 -> 40.6 | No clear effect |
| Apply SAM to both cross-attention directions instead of vision-to-text only (ablation) | TiCA · THUMOS14 50%-50% unseen | mAP@Avg: 41.2 -> 40.7 | No clear effect |
| Replace the Gaussian SAM kernel with Tophat or Cauchy (ablation) | Ti-FAD · THUMOS14 50%-50% unseen | mAP@Avg: 41.2; 40.9; 41.1 | No clear effect |
Surprising findings
- Text-to-vision and vision-to-text cross-attention with the SAM bias perform similarly, suggesting the mask itself rather than the attention direction matters most.
- The cross-modal baseline without TPT already outperforms prior foreground-based methods in the 50%-50% setting on THUMOS14.
Limitations
- Scalability to larger datasets and real-time processing not explored.
- Only evaluated on THUMOS14 and ActivityNet v1.3 with I3D and CLIP features.
- Baseline underperforms ZEETAD in the 75%-25% setting on ActivityNet-style comparisons without the Ti-FAD components.
Experimental setup
Datasets: THUMOS14 ActivityNet v1.3
Benchmarks: THUMOS14 ActivityNet v1.3
Metrics: mAP average mAP over tIoU thresholds
Baselines: STALE ZEETAD EffPrompt UnLoc B-I B-II
Builds on: ActionFormer CLIP UnLoc EffPrompt
Closest papers by meaning
- Building a Multi-modal Spatiotemporal Expert for Zero-shot Action Recognition with CLIP (AAAI 2025 · similarity 0.88)
- Test-Time Zero-Shot Temporal Action Localization (CVPR 2024 · similarity 0.88)
- Does Video-Text Pretraining Help Open-Vocabulary Online Action Detection? (NeurIPS 2024 · similarity 0.88)
- Dual DETRs for Multi-Label Temporal Action Detection (CVPR 2024 · similarity 0.87)
- Kronecker Mask and Interpretive Prompts are Language-Action Video Learners (ICLR 2025 · similarity 0.86)
- FACT: Frame-Action Cross-Attention Temporal Modeling for Efficient Action Segmentation (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.