Research map / Video / Action recognition
Prediction-Feedback DETR for Temporal Action Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Pred-DETR, a DETR-based temporal action detector that identifies attention collapse in cross-attention and uses IoU relations of predictions (decoder and encoder) as KL-divergence feedback targets to restore cross- and self-attention diversity.
Problem
DETR-based temporal action detection suffers from attention collapse, especially in cross-attention, which degrades localization performance.
Approach
Align cross-attention and self-attention query/key relations with the pairwise temporal-IoU similarity of the model's own predictions (decoder and encoder), used as auxiliary KL objectives, instead of relying on the collapsed cross-attention as guidance as in Self-DETR.
Key results
- Pred-DETR with VideoMAEv2-g features reaches 68.5 average mAP on THUMOS14, versus 56.7 for Self-DETR-style DETR baselines with I3D.
- With I3D features, Pred-DETR achieves 36.00 average mAP on ActivityNet-v1.3 (38.62 with UniFormer-v2 classification), and 33.3 on HACS versus 31.8 for reproduced Self-DETR.
- On FineAction, Pred-DETR reaches 13.4 average mAP versus 10.7 for reproduced Self-DETR; diversity of attention maps no longer collapses with depth.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add prediction-feedback objectives (SA encoder, SA decoder, CA decoder) to DETR (ablation) | DAB-DETR-based TAD (enhanced) · ActivityNet-v1.3 | Avg. mAP (0.5-0.95): 34.14 -> 38.62 | Helped |
| add decoder cross-attention prediction-feedback (L_CA^d) (ablation) | DAB-DETR-based TAD · ActivityNet-v1.3 | Avg. mAP (0.5-0.95): 34.14 -> 38.14 | Helped |
| add encoder self-attention prediction-feedback (L_SA^e) (ablation) | DAB-DETR-based TAD · ActivityNet-v1.3 | Avg. mAP (0.5-0.95): 34.14 -> 36.67 | Helped |
| add decoder self-attention prediction-feedback (L_SA^d) (ablation) | DAB-DETR-based TAD · ActivityNet-v1.3 | Avg. mAP (0.5-0.95): 34.14 -> 35.11 | Helped |
| use prediction relation (IoU) rather than CA-derived relation as self-attention target (ablation) | Pred-DETR without CA feedback · ActivityNet-v1.3 | Avg. mAP (0.5-0.95): 36.18 -> 37.65 | Helped |
| use ground-truth or prediction intervals directly as CA target instead of indirect prediction relation (ablation) | Pred-DETR CA feedback · ActivityNet-v1.3 | Avg. mAP (0.5-0.95): 38.62 -> 34.12 (GT) / 34.62 (pred intervals) | Hurt |
| apply Pred-DETR vs original-DETR/Self-DETR with standard attention | DETR for TAD · THUMOS14 / ActivityNet-v1.3 / HACS / FineAction | Avg. mAP: e.g. HACS 31.8 -> 37.4; FineAction 10.7 -> 13.4 | Helped |
| switch backbone features from I3D to VideoMAEv2-g on THUMOS14 | Pred-DETR · THUMOS14 | Avg. mAP (0.3-0.7): 61.6 -> 68.5 | Helped |
| add stable matching to enhanced DAB-DETR (ablation) | DAB-DETR for TAD · ActivityNet-v1.3 | Avg. mAP (0.5-0.95): 31.08 -> 33.15 | Helped |
| add hybrid matching on top of stable matching (ablation) | DAB-DETR for TAD · ActivityNet-v1.3 | Avg. mAP (0.5-0.95): 33.15 -> 34.10 | Helped |
| add two-stage mechanism on top of stable and hybrid matching (ablation) | DAB-DETR for TAD · ActivityNet-v1.3 | Avg. mAP (0.5-0.95): 34.10 -> 34.14 | No clear effect |
Surprising findings
- Cross-attention in DETR for TAD is itself collapsed, contrary to the assumption in Self-DETR that cross-attention is a sound guidance signal.
- Directly matching cross-attention to ground-truth or predicted intervals degrades performance, while the indirect query-relation (IoU) target improves it.
Limitations
- DETR-based models still underperform non-DETR methods such as ActionFormer and TriDet on short-action benchmarks like THUMOS14 and FineAction.
- Predictions from undertrained models may be noisy early in training; feedback losses do not converge to zero.
- Encoder-side feedback relies on extra encoder prediction heads used only during training.
Experimental setup
Datasets: THUMOS14 ActivityNet-v1.3 HACS FineAction Kinetics
Benchmarks: THUMOS14 ActivityNet-v1.3 HACS FineAction
Metrics: mAP at tIoU thresholds average mAP attention diversity d(A)
Baselines: BMN G-TAD AFSD TAGS ActionFormer TriDet DyFaDet RTD-Net TadTR ReAct Self-DETR TCANet
Builds on: DETR DAB-DETR Self-DETR Deformable DETR TadTR
Closest papers by meaning
- Dual DETRs for Multi-Label Temporal Action Detection (CVPR 2024 · similarity 0.88)
- End-to-End Temporal Action Detection with 1B Parameters Across 1000 Frames (CVPR 2024 · similarity 0.87)
- Text-Infused Attention and Foreground-Aware Modeling for Zero-Shot Temporal Action Detection (NeurIPS 2024 · similarity 0.85)
- Adapting Short-Term Transformers for Action Detection in Untrimmed Videos (CVPR 2024 · similarity 0.85)
- Benchmarking the Robustness of Temporal Action Detection Models Against Temporal Corruptions (CVPR 2024 · similarity 0.84)
- Context-Enhanced Memory-Refined Transformer for Online Action Detection (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.