Research map / Video / Action recognition
Temporal Action Localization with Cross Layer Task Decoupling and Refinement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CLTDR-GMG, a one-stage temporal action localization model with a gated multi-granularity encoder (GMG) and a cross-layer task decoupling and refinement decoder (CLTDR), reporting state-of-the-art mAP on five TAL benchmarks.
Problem
Temporal action localization requires classification and boundary regression, which have conflicting feature requirements; existing one-stage methods decouple the two tasks only at the parameter level on shared features and mostly lack global temporal context.
Approach
Decouple classification and regression by fusing features from higher pyramid layers (semantic) for classification and lower layers (boundary detail) for regression via attention, refine both outputs with cross-layer features, and capture instant, local and global temporal context with a gated three-branch GMG block that uses FFT-based global filtering instead of self-attention.
Key results
- On THUMOS14 the method reaches 69.9% avg mAP with I3D features, 71.8% with VideoMAEv2, and 74.3% with InternVideo2-6B, at least 0.6/1.7/1.3 points above prior SOTA.
- On MultiTHUMOS with VideoMAEv2 features it reaches 44.9% avg mAP, versus 37.5% for TriDet.
- Ablation on THUMOS14 shows the gated mechanism adds 0.4 avg mAP (73.9 to 74.3) and FFT global filter matches self-attention (74.3 vs 74.2) with about 2.7x fewer parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add GMG instant, local, and global granularity branches (ablation) | CLTDR-GMG · THUMOS14 | average mAP: 73.9 -> 74.3 | Helped |
| Add the gated mechanism to GMG (ablation) | CLTDR-GMG · THUMOS14 | average mAP: 73.9 -> 74.3 | Helped |
| Replace FFT with self-attention for GMG global feature extraction (ablation) | CLTDR-GMG · THUMOS14 | average mAP: 74.3 -> 74.2 | No clear effect |
| Use a higher pyramid layer for classification features (ablation) | CLTDR classification module · THUMOS14 | mAP: 73.3 -> 74.3 | Helped |
| Use a lower pyramid layer for regression features (ablation) | CLTDR regression module · THUMOS14 | mAP: 73.4 -> 74.3 | Helped |
| Use both higher and lower pyramid layers for classification and regression (ablation) | CLTDR · THUMOS14 | mAP: 74.3 -> 74.0; 74.3 -> 73.2 | Hurt |
| Add RefineHead for classification and regression (ablation) | CLTDR · THUMOS14 | average mAP: 73.5 -> 74.3 | Helped |
| Use attention instead of concatenation or addition for cross-layer fusion (ablation) | CLTDR · THUMOS14 | average mAP: 73.4 -> 74.3; 73.5 -> 74.3 | Helped |
| Increase feature pyramid levels from 1 to 6 (ablation) | CLTDR-GMG · THUMOS14 | average mAP: 55.6 -> 74.3 | Helped |
| Increase feature pyramid levels from 6 to 7 or 8 (ablation) | CLTDR-GMG · THUMOS14 | average mAP: 74.3 -> 73.6; 74.3 -> 73.7 | Hurt |
| Use CLTDR-GMG instead of TriDet | THUMOS14 with InterVideo2-6B · THUMOS14 | average mAP: 73.0 -> 74.3 | Helped |
| Use VideoMAEv2 instead of I3D RGB features | CLTDR-GMG · MultiTHUMOS | average mAP: 42.7 -> 44.9 | Helped |
Surprising findings
- Using both higher and lower pyramid layers together for classification and regression degraded performance slightly compared with using one neighbouring layer, suggesting excess cross-layer information can be detrimental.
Limitations
- Gains from individual components are modest (often 0.4-0.8 mAP), and results depend on strong pretrained features such as InternVideo2-6B.
- Localization error and background confusion remain the dominant error types according to the error diagnosis.
Experimental setup
Datasets: THUMOS14 MultiTHUMOS EPIC-KITCHENS-100 ActivityNet-1.3 HACS
Benchmarks: THUMOS14 MultiTHUMOS EPIC-KITCHENS-100 ActivityNet-1.3 HACS
Metrics: mAP average mAP over tIoU thresholds
Baselines: ActionFormer TemporalMaxer TriDet TFFormer TransGMC ActionMamba TALLFormer PointTAD TadTR ReAct TCANet
Builds on: ActionFormer TriDet Trident-head TSCODE FNet
Closest papers by meaning
- Learning Disentangled Classification and Localization Representations for Temporal Action Localization (AAAI 2022 · similarity 0.91)
- Low-Fidelity Video Encoder Optimization for Temporal Action Localization (NeurIPS 2021 · similarity 0.91)
- Weakly Supervised Temporal Action Localization Through Learning Explicit Subspaces for Action and Context (AAAI 2021 · similarity 0.90)
- Test-Time Zero-Shot Temporal Action Localization (CVPR 2024 · similarity 0.90)
- Energy vs. Noise: Towards Robust Temporal Action Localization in Open-World (AAAI 2025 · similarity 0.88)
- Realigning Confidence with Temporal Saliency Information for Point-Level Weakly-Supervised Temporal Action Localization (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.