Research map / Video / Action recognition
Learning to Predict Activity Progress by Self-Supervised Video Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Generalizes temporal cycle-consistency (TCC) with GMM-based multi-neighbor cycle-back loss, a context-dependent drop function, and a multi-head crosstask network for self-supervised video alignment and online progress prediction.
Problem
Self-supervised video alignment and activity progress prediction on in-the-wild videos, which contain repeated actions, background/redundant frames, and varying action orderings, and where existing methods train per-task models and need task labels at inference.
Approach
Replace TCC's single soft nearest neighbor with multiple GMM-derived soft nearest neighbors and a multi-cycle-back regression loss, weight alignment loss by a learned context-conditioned drop probability for background frames, and share one multi-head network across tasks so no activity label is needed at inference.
Key results
- On Penn Action, GTCC reduces online progress error (OPE) from 25.5% (LAV) to 14.5% and improves KT by at least 7.8% over other methods.
- On EgoProceL, GTCC improves PC (1.0) to 80.6 vs 68.3 for TCC and reduces OPE from 34.0% (VAVA) to 13.6%.
- On COIN, GTCC drop accuracy reaches 62.0% versus 44.7% for VAVA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GTCC instead of LAV | video alignment and progress prediction · Pouring | EAE: 9.0 -> 7.9 | Helped |
| Use GTCC instead of LAV | video alignment and progress prediction · Penn Action | OPE: 25.5 -> 14.5 | Helped |
| Use GTCC instead of LAV | video alignment and progress prediction · EgoProceL | EAE: 20.8 -> 13.3 | Helped |
| Use GTCC instead of LAV | video alignment and progress prediction · COIN | EAE: 26.7 -> 18.9 | Helped |
| Add the context-dependent drop term (ablation) | GTCC · EgoProceL | EAE: 24.3 -> 13.3 | Helped |
| Add the context-dependent drop term (ablation) | GTCC · COIN | PC (1.0): 46.6 -> 56.3 | Helped |
| Use 5 MCN heads instead of 1 (ablation) | GTCC on CMU-MMAC · CMU-MMAC | EAE: 29.5 -> 14.9 | Helped |
| Use 5 MCN heads instead of 1 (ablation) | GTCC on EGTEA · EGTEA | EAE: 15.5 -> 10.2 | Helped |
| Increase GMM components K from 1 to 15 (ablation) | GTCC · COIN | EAE: 23.2 -> 18.9 | Helped |
| Increase GMM components K from 1 to 15 (ablation) | GTCC · Penn Action | EAE: 26.7 -> 16.1 | Helped |
Surprising findings
- With a single ResNet-50 embedding in the multi-task setting, all methods perform generally poorly, and the MCN architecture is needed for gains.
- GTCC does not improve phase classification (PC) on monotonic datasets relative to baselines, since PC does not measure fine alignment.
Limitations
- GTCC underperforms on some metrics such as PC on monotonic datasets and OPE on some multi-task settings.
- Single-task setting requires training a separate model per activity.
Experimental setup
Datasets: Pouring Penn Action COIN EgoProceL CMU-MMAC EGTEA Gaze+
Benchmarks: Pouring Penn Action COIN EgoProceL CMU-MMAC EGTEA Gaze+
Metrics: Phase Classification Enclosed Area Error Kendall's Tau Kendall's Tau in the Wild PHP Online Progress Error precision recall F1 accuracy
Baselines: SAL TCN TCC LAV VAVA
Builds on: TCC VAVA LAV Drop-DTW
Closest papers by meaning
- Temporal Cycle-Consistency Learning (CVPR 2019 · similarity 0.91)
- Learning To Align Sequential Actions in the Wild (CVPR 2022 · similarity 0.88)
- Learning by Aligning Videos in Time (CVPR 2021 · similarity 0.87)
- Weakly-Guided Self-Supervised Pretraining for Temporal Activity Detection (AAAI 2023 · similarity 0.86)
- Representation Learning via Global Temporal Alignment and Cycle-Consistency (CVPR 2021 · similarity 0.86)
- Frame-Wise Action Representations for Long Videos via Sequence Contrastive Learning (CVPR 2022 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.