Research map / Video / Video segmentation and tracking
Collaborative Weakly Supervised Video Correlation Learning for Procedure-Aware Instructional Video Analysis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Collaborative Procedure Alignment (CPA), a weakly supervised framework that jointly mines step segmentations for paired instructional videos via dynamic programming and aligns frames to steps to measure video correlation, evaluated on sequence verification and action quality assessment.
Problem
Procedure-aware correlation learning on instructional videos normally requires costly step-level annotations (step labels and boundaries); the paper targets learning procedural knowledge using only video-level class labels.
Approach
Collaborative step mining uses dynamic programming over the frame-to-frame similarity matrix of paired videos to produce consistent step segmentations for both videos simultaneously, and frame-to-step alignment computes a DP-based alignment probability (negative log-likelihood as distance) in both directions to measure procedural correlation.
Key results
- Sequence verification AUC of 88.14% on CSV and 88.11% on Diving-SV with CPA+X3D, the best among compared methods on those datasets.
- On FineDiving, CPA+TSA reaches Spearman's rho 0.9364 and R-l2 0.2909, outperforming TSA (our impl.) 0.9302 / 0.3154 without step-level annotations.
- Ablation on CSV shows the baseline AUC 81.65 rising to 86.23 with frame-to-step alignment and to 88.14 with collaborative step mining added.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CPA+R50 instead of the strongest listed baseline | Sequence verification · CSV | AUC: 86.92 -> 88.14 | Helped |
| Use CPA+X3D instead of the strongest listed baseline | Sequence verification · Diving-SV | AUC: 86.09 -> 88.11 | Helped |
| Use CPA+R50 instead of WeakSVR | Sequence verification · COIN-SV | AUC: 59.57 -> 57.57 | Hurt |
| Replace supervised TSA segmentation with CPA segmentation | TSA on FineDiving · FineDiving | AIoU@0.5: 93.23 -> 94.28 | Helped |
| Replace supervised TSA segmentation with CPA segmentation | TSA on FineDiving · FineDiving | AIoU@0.75: 53.39 -> 21.14 | Hurt |
| Replace supervised TSA segmentation with CPA segmentation | TSA on FineDiving · FineDiving | Spearman's ρ: 0.9302 -> 0.9364 | Helped |
| Replace supervised TSA segmentation with CPA segmentation | TSA on FineDiving · FineDiving | R-ℓ2 (× 100): 0.3154 -> 0.2909 | Helped |
| Add FSA to the video-level feature verification setup (ablation) | CPA on CSV · CSV | AUC: 81.65 -> 86.23 | Helped |
| Add CSM to FSA with uniformly sampled step features (ablation) | CPA on CSV · CSV | AUC: 86.23 -> 88.14 | Helped |
| Add FSA to uniformly divided video segments (ablation) | CPA on FineDiving · FineDiving | Spearman's ρ: 0.9221 -> 0.9275 | Helped |
| Add FSA to uniformly divided video segments (ablation) | CPA on FineDiving · FineDiving | R-ℓ2: 0.3456 -> 0.3288 | Helped |
| Add CSM to FSA with uniformly sampled step features (ablation) | CPA on FineDiving · FineDiving | Spearman's ρ: 0.9275 -> 0.9364 | Helped |
Surprising findings
- On AIoU@0.75, CPA+TSA (21.14) is far below TSA (our impl.) (53.39), despite its better AQA ranking results.
Limitations
- Step number K must be chosen per dataset; performance peaks at dataset-specific values and drops otherwise.
- Segmentation quality at finer IoU thresholds is weaker than the TSA baseline (AIoU@0.75 21.14 vs 53.39).
Experimental setup
Datasets: CSV Diving-SV COIN-SV FineDiving Kinetics-400
Benchmarks: CSV Diving-SV COIN-SV FineDiving
Metrics: AUC Spearman's rho relative L2 distance AIoU
Baselines: TRN TSM Video-Swin CAT OTAM TAP Drop-DTW CLIP+TE+MLP WeakSVR USDL MUSDL CoRe
Builds on: TSA OTAM Drop-DTW
Closest papers by meaning
- Learning Procedure-Aware Video Representation From Instructional Videos and Their Narrations (CVPR 2023 · similarity 0.88)
- HT-Step: Aligning Instructional Articles with How-To Videos (NeurIPS 2023 · similarity 0.88)
- Weakly-Supervised Online Action Segmentation in Multi-View Instructional Videos (CVPR 2022 · similarity 0.88)
- Masked Temporal Interpolation Diffusion for Procedure Planning in Instructional Videos (ICLR 2025 · similarity 0.84)
- Detours for Navigating Instructional Videos (CVPR 2024 · similarity 0.84)
- Step Differences in Instructional Video (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.