Research map / Video / Action recognition
Action-Agnostic Point-Level Supervision for Temporal Action Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes action-agnostic point-level (AAPL) supervision, where unsupervised-sampled frames are labeled with action classes, plus a scoring model and losses (point, video, prototype contrastive, pseudo-labeling) for temporal action detection.
Problem
Temporal action detection needs costly annotations; video-level supervision is weak for localization and point-level supervision requires annotators to search for every action instance, which is expensive.
Approach
Annotators label only frames sampled without human intervention (e.g., at regular intervals) with action categories, including background frames, avoiding the search for every instance; a detector is trained with point loss, an incomplete-label-aware video loss (top-k pooling), a prototype-anchored supervised contrastive loss, and ground-truth anchored pseudo-labeling.
Key results
- On GTEA and BEOID, AAPL (3 sec.) reaches 46.3 and 55.2 average mAP, above the best point-level method's 44.9 and 53.3.
- On FineAction, AAPL with 30-second intervals reaches 6.0 average mAP versus 3.5 for the point-level baseline reproduced by the authors.
- On ActivityNet 1.3, AAPL at 10-second intervals reaches 25.7 average mAP versus 25.1 for the point-level method LACP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use AAPL labels sampled at 3-second intervals instead of point-level supervision | Temporal action detection · BEOID | Avg mAP: 53.3 -> 55.2 | Helped |
| use AAPL labels sampled at 3-second intervals instead of point-level supervision | Temporal action detection · GTEA | Avg mAP: 44.9 -> 46.3 | Helped |
| use AAPL labels sampled at 30-second intervals instead of video-level supervision | Temporal action detection · FineAction | Avg mAP: 4.1 -> 6.0 | Helped |
| use AAPL labels sampled at 10-second intervals instead of point-level supervision | Temporal action detection · ActivityNet 1.3 | Avg mAP: 25.1 -> 25.7 | Helped |
| increase AAPL sampling interval from 10 to 30 seconds (ablation) | Temporal action detection · ActivityNet 1.3 | Avg mAP: 25.7 -> 24.7 | Hurt |
| add the video loss to the point loss (ablation) | AAPL training · THUMOS '14 (30 sec. AAPL) | Avg mAP: 40.3 -> 42.8 | Helped |
| add the video loss to the point loss (ablation) | AAPL training · BEOID (3 sec. AAPL) | Avg mAP: 32.0 -> 46.2 | Helped |
| add the prototype-anchored supervised contrastive loss (ablation) | AAPL training with point and video losses · THUMOS '14 (30 sec. AAPL) | Avg mAP: 42.8 -> 43.6 | Helped |
| add the prototype-anchored supervised contrastive loss (ablation) | AAPL training with point and video losses · BEOID (3 sec. AAPL) | Avg mAP: 46.2 -> 54.1 | Helped |
| add ground-truth anchored pseudo-labeling (ablation) | AAPL training with the full objective · THUMOS '14 (30 sec. AAPL) | Avg mAP: 43.6 -> 44.3 | Helped |
| add ground-truth anchored pseudo-labeling (ablation) | AAPL training with the full objective · BEOID (3 sec. AAPL) | Avg mAP: 54.1 -> 55.2 | Helped |
| replace the proposed video loss with BCE loss (ablation) | Point-loss AAPL training · THUMOS '14 (30 sec. AAPL) | Avg mAP: 42.8 -> 40.8 | Hurt |
Surprising findings
- LACP, the strongest point-level method on THUMOS'14, underperforms HAAN on FineAction, which the authors attribute to sparse action instances.
- A simple baseline with point loss only and 30-second AAPL intervals on THUMOS'14 is comparable to a prior point-level method while costing about one-third of the annotation time.
Limitations
- Action-agnostic sampling may miss some action instances or entire action classes, so labels can be incomplete.
- Annotation-time measurements use a modified VIA tool and eight workers, and may not reflect real annotation pipelines, especially for many classes.
- Video loss adapted for incomplete labels; for datasets with many classes annotation is unrealistic under the implemented pipeline.
Experimental setup
Datasets: BEOID GTEA THUMOS'14 FineAction ActivityNet 1.3
Benchmarks: BEOID GTEA THUMOS'14 FineAction ActivityNet 1.3
Metrics: mAP at temporal IoU thresholds average mAP annotation time relative to video duration
Baselines: LACP HAAN video-level methods (e.g., W-TALC/feature-erasing/attention-based methods) full supervision annotation cost
Builds on: LACP HAAN SupCon loss focal loss soft non-maximum suppression
Closest papers by meaning
- Action Sets: Weakly Supervised Action Segmentation Without Ordering Constraints (CVPR 2018 · similarity 0.88)
- Action Recognition From Single Timestamp Supervision in Untrimmed Videos (CVPR 2019 · similarity 0.88)
- PointTAD: Multi-Label Temporal Action Detection with Learnable Query Points (NeurIPS 2022 · similarity 0.88)
- Are all Frames Equal? Active Sparse Labeling for Video Action Detection (NeurIPS 2022 · similarity 0.87)
- End-to-End Temporal Action Detection with 1B Parameters Across 1000 Frames (CVPR 2024 · similarity 0.85)
- Stable Mean Teacher for Semi-supervised Video Action Detection (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.