Research map / Video / Action recognition
Boosting Point-Supervised Temporal Action Localization through Integrating Query Reformation and Optimal Transport
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes QROT, an end-to-end DETR-based framework for point-supervised temporal action localization that uses clustered semantic consensus queries and optimal-transport pseudo-labels for Hungarian matching.
Problem
Point-supervised temporal action localization with only one timestamp per action; MIL-based methods struggle with global context and need heuristic post-processing, while naively adapting DETR fails because point labels are not encoded into queries and no complete segments exist for matching decoder
Approach
Initialize DETR decoder queries from K-means cluster centers of point-labeled snippet features per class (semantic consensus queries), and generate segment-level pseudo labels by formulating point-to-snippet assignment as an optimal transport problem solved with Sinkhorn-Knopp, which then drive Hungarian matching of decoder proposals.
Key results
- On THUMOS14, QROT reaches 52.1% average mAP@[0.3:0.7] and 72.3% average mAP@[0.1:0.5], versus 51.1% for HR-Pro under point supervision.
- On ActivityNet-v1.3, QROT achieves 27.7% average mAP, slightly above SMBD (27.5%) and HR-Pro (27.1%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use QROT instead of HR-Pro | Point-supervised temporal action localization · THUMOS14 test | mAP@[0.3:0.7]: 51.1 -> 52.1 | Helped |
| Use QROT instead of SMBD | Point-supervised temporal action localization · ActivityNet-v1.3 | average mAP@[0.5:0.95:0.05]: 27.5 -> 27.7 | Helped |
| Replace extended point-label pseudo segments with optimal-transport pseudo labels (ablation) | Point-DETR · THUMOS14 | mAP@[0.3:0.7]: 2.2 -> 22.9 | Helped |
| Add Dynamic Label Refinement to optimal-transport pseudo-label generation (ablation) | Point-DETR with optimal transport · THUMOS14 | mAP@[0.3:0.7]: 22.9 -> 23.4 | Helped |
| Add Query Reformation (ablation) | Point-DETR with action loss · THUMOS14 | mAP@[0.3:0.7]: 50.9 -> 52.1 | Helped |
Limitations
- Improvements over the best prior point-supervised methods are modest (e.g., ~1% average mAP on THUMOS14, 0.2% on ActivityNet).
- Pseudo-label generation relies on classification scores in early training, requiring a frozen decoder for the first 6 epochs.
- Number of static queries (clusters per class) is fixed at three and not analyzed in depth.
Experimental setup
Datasets: THUMOS14 ActivityNet-v1.3
Benchmarks: THUMOS14 ActivityNet-v1.3
Metrics: mAP@IoU average mAP
Baselines: SF-Net Ju et al. LACP HR-Pro RCTSI SMBD BSN BMN G-TAD React ASL
Builds on: DETR Sinkhorn-Knopp SF-Net LACP HR-Pro
Closest papers by meaning
- HR-Pro: Point-Supervised Temporal Action Localization via Hierarchical Reliability Propagation (AAAI 2024 · similarity 0.86)
- Temporal Action Localization by Structured Maximal Sums (CVPR 2017 · similarity 0.84)
- PointTAD: Multi-Label Temporal Action Detection with Learnable Query Points (NeurIPS 2022 · similarity 0.84)
- Realigning Confidence with Temporal Saliency Information for Point-Level Weakly-Supervised Temporal Action Localization (CVPR 2024 · similarity 0.84)
- Bridge the Gap: From Weak to Full Supervision for Temporal Action Localization with PseudoFormer (CVPR 2025 · similarity 0.84)
- Weakly-Supervised Temporal Action Localization by Inferring Salient Snippet-Feature (AAAI 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.