Research map / Detection and segmentation / CNN features
Dual DETRs for Multi-Label Temporal Action Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DualDETR, a dual-level (instance and boundary) query-based temporal action detection framework with two-branch decoding and joint query initialization, achieving state-of-the-art det-mAP on multi-label TAD benchmarks.
Problem
Query-based temporal action detection (DETR-style) predicts actions only at the instance level via center points, giving sub-optimal boundary localization, especially in multi-label settings with overlapping actions.
Approach
Decodes actions at both instance and boundary (start/end) levels with two separate decoder branches over channel-partitioned encoder features, aligns one-to-one dual-level queries to encoder proposals, and initializes queries' position and content vectors from matched proposals, with a mutual refinement module exchanging position information.
Key results
- Achieves 32.6% avg det-mAP on MultiTHUMOS (vs. 23.5% for PointTAD) and 15.3% det-mAP on Charades (vs. 12.1%).
- Reaches 20.8% det-mAP on TSU, versus 10.6% for MS-TCT, with 5.5 GFLOPs.
- Ablation: two-branch design adds about 2%, query alignment about 1.5%, and joint initialization about 2.1% avg det-mAP on MultiTHUMOS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add two-branch decoding to the combined instance- and boundary-level framework (ablation) | DualDETR · MultiTHUMOS | DetmAP average: 27.04 -> 29.04 | Helped |
| add query alignment between dual-level queries and encoder proposals (ablation) | DualDETR with two-branch decoding · MultiTHUMOS | DetmAP average: 29.04 -> 30.55 | Helped |
| add joint query initialization using matched proposal priors (ablation) | DualDETR with query alignment · MultiTHUMOS | DetmAP average: 30.55 -> 32.64 | Helped |
| use instance-level detection only instead of dual-level detection (ablation) | DualDETR · MultiTHUMOS | DetmAP average: 32.64 -> 28.62 | Hurt |
| use boundary-level detection only instead of dual-level detection (ablation) | DualDETR · MultiTHUMOS | DetmAP average: 32.64 -> 22.92 | Hurt |
| replace parallel refinement with sequential boundary-then-instance refinement (ablation) | DualDETR mutual refinement · MultiTHUMOS | DetmAP average: 32.64 -> 32.35 | Hurt |
| replace parallel refinement with sequential instance-then-boundary refinement (ablation) | DualDETR mutual refinement · MultiTHUMOS | DetmAP average: 32.64 -> 32.50 | Hurt |
| update position vectors only at the last decoder layer rather than at each layer (ablation) | DualDETR mutual refinement · MultiTHUMOS | DetmAP average: 32.64 -> 31.81 | Hurt |
| also refine content vectors during mutual refinement (ablation) | DualDETR mutual refinement · MultiTHUMOS | DetmAP average: 32.64 -> 31.82 | Hurt |
| use DualDETR rather than PointTAD | I3D-based TAD on MultiTHUMOS · MultiTHUMOS | DetmAP: 23.5 -> 32.6 | Helped |
| use DualDETR rather than MS-TCT | I3D-based TAD on Charades · Charades | SegmAP: 25.4 -> 23.2 | Hurt |
Surprising findings
- Simply decoding instance and boundary queries with a shared decoder performs worse than the two-branch design, and boundary-only detection is much weaker than instance-only (22.9 vs 28.6 avg det-mAP).
Limitations
- Segmentation-mAP on Charades and MultiTHUMOS is not uniformly best (e.g., CTRN is higher on MultiTHUMOS seg-mAP at 44.0 vs 45.5 comparison is favorable but Charades seg-mAP trails CTRN 25.3 vs 23.2).
- Relies on offline pre-extracted I3D features with fixed windowing rather than end-to-end training.
Experimental setup
Datasets: MultiTHUMOS Charades TSU
Benchmarks: MultiTHUMOS Charades TSU
Metrics: detection-mAP segmentation-mAP GFLOPs
Baselines: PointTAD MS-TCT PDAN CTRN MLAD Coarse-Fine TadTR ActionFormer TriDet AFSD
Builds on: DETR Deformable DETR DAB-DETR DINO RTDNet PointTAD
Closest papers by meaning
- TE-TAD: Towards Full End-to-End Temporal Action Detection via Time-Aligned Coordinate Expression (CVPR 2024 · similarity 0.89)
- DiGIT: Multi-Dilated Gated Encoder and Central-Adjacent Region Integrated Decoder for Temporal Action Detection Transformer (CVPR 2025 · similarity 0.89)
- End-to-End Temporal Action Detection with 1B Parameters Across 1000 Frames (CVPR 2024 · similarity 0.88)
- Prediction-Feedback DETR for Temporal Action Detection (AAAI 2025 · similarity 0.88)
- Text-Infused Attention and Foreground-Aware Modeling for Zero-Shot Temporal Action Detection (NeurIPS 2024 · similarity 0.87)
- Benchmarking the Robustness of Temporal Action Detection Models Against Temporal Corruptions (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.