Research map / Detection and segmentation / CNN features
DiGIT: Multi-Dilated Gated Encoder and Central-Adjacent Region Integrated Decoder for Temporal Action Detection Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiGIT, a temporal action detection transformer with a multi-dilated gated encoder (MDGE) and a central-adjacent region integrated decoder (CAID), achieving state-of-the-art on THUMOS14, ActivityNet v1.3, and HACS-Segment.
Problem
Query-based temporal action detectors inherit object-detection architectures, causing redundant multi-scale features in the encoder and center-focused deformable sampling in the decoder that miss surrounding temporal context.
Approach
Replaces multi-scale deformable encoder with single-scale multi-dilated gated convolutions (dilation rates 1..Nd with SiLU gating), and adds an adjacent-region cross-attention after central-region cross-attention in the decoder by re-initializing sampling offset biases to left/right of the center.
Key results
- DiGIT with InternVideo2 features reaches 73.8 average mAP on THUMOS14 (0.3:0.7), vs 70.3 for TE-TAD.
- DiGIT achieves 45.9 average mAP on HACS-Segment with InternVideo2, vs 44.1 for TE-TAD.
- Applying MDGE and CAID to TadTR improves its THUMOS14 average mAP from 67.3 to 69.9 with InternVideo2 features.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace TE-TAD with DiGIT | TE-TAD with InternVideo2 features · THUMOS14 | mAP (Avg.): 70.3 -> 73.8 | Helped |
| Replace TE-TAD with DiGIT | TE-TAD with I3D / R(2+1)D features · THUMOS14 | mAP (Avg.): 67.9 -> 69.0 | Helped |
| Replace ActionMamba with DiGIT | InternVideo2 features · ActivityNet v1.3 | mAP (Avg.): 42.0 -> 42.0 | No clear effect |
| Replace DyFADet with DiGIT | InternVideo2 features · HACS-Segment | mAP (Avg.): 44.3 -> 45.9 | Helped |
| Add adjacent-region cross-attention after central-region cross-attention (ablation) | CAID on THUMOS14 · THUMOS14 | mAP (Avg.): 72.5 -> 73.8 | Helped |
| Use adjacent-region cross-attention alone instead of central-region cross-attention (ablation) | Decoder on THUMOS14 · THUMOS14 | mAP (Avg.): 72.5 -> 69.2 | Hurt |
| Expand the CCA initial sampling range by 1.5× (ablation) | Decoder on THUMOS14 · THUMOS14 | mAP (Avg.): 72.5 -> 72.6 | No clear effect |
| Expand the CCA initial sampling range by 2.0× (ablation) | Decoder on THUMOS14 · THUMOS14 | mAP (Avg.): 72.5 -> 71.8 | Hurt |
| Add a second central-region cross-attention layer (ablation) | Decoder on THUMOS14 · THUMOS14 | mAP (Avg.): 72.5 -> 72.7 | No clear effect |
| Add MDGE (ablation) | TadTR on THUMOS14 · THUMOS14 | mAP (Avg.): 67.3 -> 68.4 | Helped |
| Add CAID (ablation) | TadTR on THUMOS14 · THUMOS14 | mAP (Avg.): 67.3 -> 67.9 | Helped |
| Add MDGE and CAID (ablation) | TadTR on THUMOS14 · THUMOS14 | mAP (Avg.): 67.3 -> 69.9 | Helped |
Surprising findings
- Expanding the central-region sampling initialization range (1.5x, 2.0x) does not improve performance, and adjacent-region cross-attention alone significantly degrades mAP (69.2).
Limitations
- Evaluated only on pre-extracted features; backbone is not fine-tuned.
- ActivityNet v1.3 results use external classification scores, and DiGIT is not uniformly best there (e.g., ActionMamba and DyFADet are higher on some columns).
Experimental setup
Datasets: THUMOS14 ActivityNet v1.3 HACS-Segment
Benchmarks: THUMOS14 ActivityNet v1.3 HACS-Segment
Metrics: mAP mAP at IoU thresholds
Baselines: TE-TAD TadTR DualDETR RTD-Net ReAct Self-DETR ActionFormer TriDet DyFADet ActionMamba AdaTAD ViT-TAD
Builds on: TE-TAD DETR Deformable DETR DINO DFM-style dilated convolution
Closest papers by meaning
- Dual DETRs for Multi-Label Temporal Action Detection (CVPR 2024 · similarity 0.89)
- TE-TAD: Towards Full End-to-End Temporal Action Detection via Time-Aligned Coordinate Expression (CVPR 2024 · similarity 0.87)
- End-to-End Temporal Action Detection with 1B Parameters Across 1000 Frames (CVPR 2024 · similarity 0.86)
- Adapting Short-Term Transformers for Action Detection in Untrimmed Videos (CVPR 2024 · similarity 0.86)
- Enhancing Temporal Understanding in Video-LLMs through Stacked Temporal Attention in Vision Encoders (NeurIPS 2025 · similarity 0.85)
- Text-Infused Attention and Foreground-Aware Modeling for Zero-Shot Temporal Action Detection (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.