Research map / Video / Video segmentation and tracking
STEPs: Self-Supervised Key Step Extraction and Localization from Unlabeled Procedural Videos
Anshul Shah, Benjamin Lundell, Harpreet Sawhney, Rama Chellappa
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add bootstrapped window (BMC2) to the contrastive loss (ablation) | STEPs · Meccano | F1: 32.1 -> 33.1 | Improved |
| add multi-cue training (RGB+optical flow) (ablation) | STEPs · Meccano | F1: 32.1 -> 34.2 | Improved |
| add both multi-cue training and BMC2 (ablation) | STEPs · Meccano | F1: 32.1 -> 36.4 | Improved |
| add multi-cue loss and BMC2 on other egocentric sets (ablation) | STEPs · EGTEA Gaze+ | F1: 30.8 -> 29.0 | No clear effect |
| use a transformer instead of LSTM or TCN temporal encoder (ablation) | STEPs · Meccano | F1: 33.6 -> 36.4 | Improved |
| train with temporal context of 1024 chunks instead of 128 (ablation) | STEPs · Meccano | F1: 33.04 -> 36.43 | Improved |
| increase training context from 1024 to 2048 chunks (ablation) | STEPs · Meccano | F1: 36.43 -> 34.88 | Hurt |
| use full-video temporal extent beta=1 instead of 0.2 (ablation) | STEPs · Meccano | F1: 24.7 -> 36.4 | Improved |
| add positional encoding to the input features (ablation) | STEPs (BMC2) · Meccano | F1: 32.2 -> 36.4 | Improved |
| use union window for both positives and negatives (ablation) | STEPs · Meccano | F1: 34.6 -> 36.4 | Improved |
| add depth as second training cue instead of gaze (ablation) | STEPs · Meccano | F1: 35.4 -> 36.6 | Improved |
| use RGB+OF at inference instead of RGB alone | STEPs · Meccano | F1: 36.4 -> 37.6 | Improved |
Later papers that compared with this method
1 later accepted paper reports STEPS as a baseline in their results tables; 1 of them report a better number on most of the results they share with it.
- OPEL: Optimal Transport Guided ProcedurE Learning (NeurIPS 2024 · better on 2 of 2 shared results)
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- StepFormer: Self-Supervised Step Discovery and Localization in Instructional Videos (CVPR 2023 · similarity 0.87)
- Learning Procedure-Aware Video Representation From Instructional Videos and Their Narrations (CVPR 2023 · similarity 0.84)
- Learning Procedural-Aware Video Representations Through State-Grounded Hierarchy Unfolding (AAAI 2026 · similarity 0.84)
- Step Differences in Instructional Video (CVPR 2024 · similarity 0.83)
- AirSketch: Generative Motion to Sketch (NeurIPS 2024 · similarity 0.83)
- VA-AR: Learning Velocity-Aware Action Representations with Mixture of Window Attention (AAAI 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
