Research map / Video / Action recognition
TE-TAD: Towards Full End-to-End Temporal Action Detection via Time-Aligned Coordinate Expression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TE-TAD, a fully end-to-end DETR-style temporal action detector that replaces normalized sigmoid coordinates with time-aligned timeline coordinates and adds adaptive query selection, removing sliding windows and NMS.
Problem
Query-based detectors for temporal action detection degrade on long/diverse-length videos and rely on sliding windows and NMS, so they cannot be fully end-to-end.
Approach
Replace normalized sigmoid-based coordinate expression with actual timeline values (length-invariant), reference-based time-aligned query generation, and sector-wise adaptive top-k query selection, so the set-prediction one-to-one matching works without NMS.
Key results
- On THUMOS14 without NMS, TE-TAD reaches 66.4% average mAP versus 56.7% for TadTR, which uses partial NMS.
- On ActivityNet v1.3 TE-TAD gets 37.0% average mAP, slightly above TadTR's 36.8%.
- On THUMOS14 TE-TAD loses only 1.5 mAP when NMS is removed (67.9 with NMS), whereas ReAct drops 35.7 points and ActionFormer 23.6.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use time-aligned coordinate expression instead of normalized coordinates (ablation) | TadTR-1 on THUMOS14 · THUMOS14 | mAP@AVG: 40.2 -> 59.5 | Helped |
| Add multi-scale features to TadTR-1 (ablation) | TadTR-1 on THUMOS14 · THUMOS14 | mAP@AVG: 40.2 -> 42.6 | Helped |
| Add the two-stage method to TadTR-1 with multi-scale features (ablation) | TadTR-1 on THUMOS14 · THUMOS14 | mAP@AVG: 42.6 -> 43.3 | Helped |
| Add adaptive query selection to the TE and multi-scale two-stage variant (ablation) | TadTR-1 on THUMOS14 · THUMOS14 | mAP@AVG: 63.6 -> 66.4 | Helped |
| Remove encoder self-attention (ablation) | TE-TAD on THUMOS14 · THUMOS14 | mAP@AVG: 61.2 without NMS; 63.8 with NMS | Hurt |
| Remove decoder self-attention (ablation) | TE-TAD on THUMOS14 · THUMOS14 | mAP@AVG: 53.0 without NMS; 63.4 with NMS | Hurt |
| Remove decoder cross-attention (ablation) | TE-TAD on THUMOS14 · THUMOS14 | mAP@AVG: 0.1 without NMS; 0.2 with NMS | Hurt |
| Remove NMS (ablation) | TE-TAD on THUMOS14 · THUMOS14 | mAP@AVG: 67.9 -> 66.4 | Hurt |
| Use time-aligned coordinate expression and adaptive query selection rather than the TadTR-1 baseline (ablation) | TE-TAD on THUMOS14 · THUMOS14 | mAP@AVG: 40.2 -> 66.4 | Helped |
| Use TE-TAD instead of TadTR | Query-based TAD on ActivityNet v1.3 · ActivityNet v1.3 | mAP@AVG: 36.8 -> 37.0 | Helped |
| Use TE-TAD instead of TadTR | Query-based TAD on THUMOS14 · THUMOS14 | mAP@AVG: 56.7 -> 66.4 | Helped |
| Use TE-TAD instead of TriDet | TAD on EpicKitchens verb · EpicKitchens verb | mAP@AVG: 25.4 -> 24.1 | Hurt |
Surprising findings
- Extending feature coverage (longer sliding windows) degrades existing query-based detectors, showing that normalized coordinates are sensitive to video length.
- Removing decoder self-attention causes a large drop without NMS (from 63.4 to 0.1 for the no-cross-attention case), indicating full-query self-attention is essential for set prediction.
Limitations
- Gains on ActivityNet v1.3 are marginal because its video lengths are less diverse.
- Still trails the strongest anchor-free detector TriDet on THUMOS14 without NMS (66.4 vs 69.3).
Experimental setup
Datasets: THUMOS14 ActivityNet v1.3 EpicKitchens
Benchmarks: THUMOS14 ActivityNet v1.3 EpicKitchens
Metrics: mAP mAP@AVG instability index false negative rate
Baselines: TadTR ReAct RTD-Net Self-DETR ActionFormer TriDet AFSD BMN G-TAD VSGN MENet TALLFormer
Builds on: TadTR DETR Deformable DETR ActionFormer focal loss
Closest papers by meaning
- End-to-End Temporal Action Detection with 1B Parameters Across 1000 Frames (CVPR 2024 · similarity 0.91)
- Dual DETRs for Multi-Label Temporal Action Detection (CVPR 2024 · similarity 0.89)
- Benchmarking the Robustness of Temporal Action Detection Models Against Temporal Corruptions (CVPR 2024 · similarity 0.87)
- Adapting Short-Term Transformers for Action Detection in Untrimmed Videos (CVPR 2024 · similarity 0.87)
- DiGIT: Multi-Dilated Gated Encoder and Central-Adjacent Region Integrated Decoder for Temporal Action Detection Transformer (CVPR 2025 · similarity 0.87)
- TIM: A Time Interval Machine for Audio-Visual Action Recognition (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.