Research map / Video / Action recognition
End-to-End Temporal Action Detection with 1B Parameters Across 1000 Frames
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AdaTAD, an end-to-end temporal action detection framework using a lightweight temporal-informative adapter (TIA) on frozen video backbones, scaling to a 1B-parameter VideoMAEv2-giant on 1,536-frame inputs.
Problem
End-to-end temporal action detection is limited by GPU memory, which restricts both backbone scale and input video length, and full fine-tuning risks forgetting and overfitting on small TAD datasets.
Approach
A temporal-informative adapter (bottleneck with temporal depth-wise convolution) inserted in a frozen backbone is the only trainable part, and an alternative placement outside the backbone (AdaTAD†) avoids backpropagating through the backbone, enabling larger models and longer videos under the same memory.
Key results
- AdaTAD with VideoMAEv2-giant on 1536x224^2 frames reaches 75.4% avg mAP on THUMOS14 (AdaTAD† variant) and 76.9% for full AdaTAD, surpassing the prior best feature-based 71.5%.
- On ActivityNet-1.3, the full AdaTAD VideoMAEv2-g model attains 41.9% avg mAP, versus 39.8% for the 768-frame variant.
- AdaTAD with VideoMAE-S uses 1M trainable backbone parameters and 2.5G memory and reaches 68.8% vs 68.4% for snippet full fine-tuning at 24.6G memory on THUMOS14.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Scale the backbone from VideoMAE-S to VideoMAEv2-g | AdaTAD · THUMOS14 | average mAP: 68.8 -> 74.8 | Helped |
| Increase input from 768 to 3072 frames (ablation) | AdaTAD with VideoMAE-S at 160² resolution · THUMOS14 | average mAP: 68.8 -> 70.6 | Helped |
| Increase frame resolution from 160² to 224² (ablation) | AdaTAD with VideoMAE-S and 768 frames · THUMOS14 | average mAP: 68.8 -> 70.7 | Helped |
| Use frame representation instead of snippet representation (ablation) | Frozen VideoMAE-S backbone · THUMOS14 | average mAP: 57.68 -> 59.35 | Helped |
| Use frame representation instead of snippet representation (ablation) | End-to-end VideoMAE-S backbone · THUMOS14 | average mAP: 68.46 -> 67.15 | Hurt |
| Replace the standard adapter with temporal depth-wise convolution in TIA (ablation) | VideoMAE-B AdaTAD · THUMOS14 | average mAP: 70.2 -> 71.5 | Helped |
| Add the depth-wise convolution residual connection (ablation) | VideoMAE-B AdaTAD · THUMOS14 | average mAP: 70.8 -> 71.5 | Helped |
| Use AdaTAD instead of full fine-tuning (ablation) | VideoMAE-S on THUMOS14 · THUMOS14 | average mAP: 67.1 -> 68.8 | Helped |
| Use external adapters and increase input from 768 to 1536 frames (ablation) | AdaTAD† with VideoMAEv2-g · THUMOS14 | average mAP: 74.6 -> 75.4 | Helped |
| Fine-tune end-to-end for TAD rather than use feature-based training (ablation) | VideoMAE-L pretrained on K400 and EPIC · EPIC-Kitchens 100 validation, Noun | average mAP: 26.98 -> 29.30 | Helped |
| Fine-tune end-to-end for TAD (ablation) | VideoMAE-L pretrained on K400 · EPIC-Kitchens 100 validation, Noun | average mAP: 15.77 -> 21.33 | Helped |
| Use end-to-end AdaTAD rather than ActionFormer feature-based detection | VideoMAE-L · EPIC-Kitchens 100 validation, Verb | average mAP: 28.7 -> 29.3 | Helped |
Surprising findings
- Frame-level representation matches or beats snippet representation in most frozen settings while using far less memory; snippet only wins end-to-end by about 1% at 8x memory.
- Adapter tuning with about 4M trainable parameters outperforms full fine-tuning of the 86M-parameter VideoMAE-B backbone (71.5 vs 70.1 mAP).
Limitations
- Full-layer adaptation of VideoMAEv2-giant at 1536 frames runs out of memory on A100-80GB, so the largest AdaTAD configuration is not feasible without the alternative placement.
- AdaTAD† incurs a slight performance drop relative to in-backbone AdaTAD.
- Experiments on EPIC-Kitchens 100 use backbones pretrained on that dataset's classification task, which limits direct comparison.
Experimental setup
Datasets: ActivityNet-1.3 THUMOS14 EPIC-Kitchens 100 Kinetics-400
Benchmarks: ActivityNet-1.3 THUMOS14 EPIC-Kitchens 100
Metrics: mAP at tIoU thresholds average mAP training memory (GB) per video
Baselines: ActionFormer TriDet E2E-TAD BasicTAD TALLFormer Re2TAL VideoMAEv2 InternVideo AFSD full fine-tuning standard adapter LongLoRA
Builds on: ActionFormer Adapter VideoMAEv2 LongLoRA TALLFormer
Closest papers by meaning
- TE-TAD: Towards Full End-to-End Temporal Action Detection via Time-Aligned Coordinate Expression (CVPR 2024 · similarity 0.91)
- Benchmarking the Robustness of Temporal Action Detection Models Against Temporal Corruptions (CVPR 2024 · similarity 0.88)
- Dual DETRs for Multi-Label Temporal Action Detection (CVPR 2024 · similarity 0.88)
- Adapting Short-Term Transformers for Action Detection in Untrimmed Videos (CVPR 2024 · similarity 0.87)
- Prediction-Feedback DETR for Temporal Action Detection (AAAI 2025 · similarity 0.87)
- HyrCap: Hybrid Rank-Calibration of Action Proposals for Temporal Event Understanding (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.