Research map / Video / Video segmentation and tracking
OPEL: Optimal Transport Guided ProcedurE Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OPEL, an optimal-transport-based procedure learning framework from multiple videos that uses Laplace-distributed optimality and temporal priors, IDM regularization, a virtual frame, and an inter-video contrastive loss, evaluated on EgoProceL, ProceL and CrossTask.
Problem
Unsupervised procedure learning: identifying key-steps and their logical order from multiple videos of the same task, without relying on monotonic alignment or frame-to-frame mapping, and handling background and redundant frames.
Approach
Treat video frames as samples from distributions and compute sequence distance via entropic optimal transport, regularized with IDM and KL-divergence terms that encode Laplace-distributed optimality and temporal priors, plus a virtual frame for background/redundant frames and an OT-guided inter-video contrastive loss.
Key results
- On EgoProceL, OPEL improves average IoU by 22.4% and F1 by 26.9% over the prior SOTA.
- On third-person benchmarks (ProceL, CrossTask), OPEL achieves 46.2% average F1 improvement over SOTA; e.g. 34.9 F1 on ProceL and 35.1 F1 on CrossTask.
- Removing both priors drops F1 and IoU by about 5 points; combined intra- and inter-video contrastive losses contribute about 3.5 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OPEL instead of GPL-w/o BG | Procedure learning on EgoProceL: MECCANO · EgoProceL: MECCANO | F1: 20.7 -> 39.2 | Helped |
| Use OPEL instead of STEPS | Procedure learning on ProceL · ProceL | F1: 24.9 -> 34.9 | Helped |
| Use OPEL instead of STEPS | Procedure learning on CrossTask · CrossTask | F1: 25.9 -> 35.1 | Helped |
| Add both temporal and optimality priors (ablation) | OPEL on MECCANO · MECCANO | F1: 34.6 -> 38.6 | Helped |
| Add the optimality prior (ablation) | OPEL on MECCANO · MECCANO | F1: ∼2 point | Helped |
| Add the temporal prior (ablation) | OPEL on MECCANO · MECCANO | F1: ∼1 point | Helped |
| Add intra- and inter-video contrastive losses (ablation) | OPEL on MECCANO · MECCANO | F1: ∼3.5 points | Helped |
| Add KL divergence as a standalone loss component (ablation) | OPEL · MECCANO | F1 | No clear effect |
| Add the virtual frame (ablation) | OPEL · MECCANO | F1 | No clear effect |
| Use graph-cut segmentation rather than K-means or subset selection (ablation) | OPEL on MECCANO · MECCANO | F1 | Helped |
| Use Laplace rather than Gaussian priors (ablation) | OPEL on MECCANO · MECCANO | F1: 33.8 -> 39.2 | Helped |
| Use k=7 rather than k=10 or higher (ablation) | OPEL on PC Assembly and PC Disassembly · PC Assembly; PC Disassembly | F1: 33.7 vs 24.3; 32.2 vs 24.8 | Helped |
Surprising findings
- The KL-divergence term as a standalone loss has minimal impact, since the IDM structure already keeps T and Q similar.
- The virtual frame has negligible individual effect, as background-heavy scenarios are not prevalent in most datasets.
Limitations
- Assumes subjects use similar objects for identical key-steps, which can cause inaccuracies when dissimilar objects are used.
- Performance drops sharply when the number of key-steps k exceeds the true number of distinct steps; k must be chosen per dataset.
Experimental setup
Datasets: EgoProceL ProceL CrossTask CMU-MMAC EGTEA-GAZE+ MECCANO EPIC-Tents
Benchmarks: EgoProceL ProceL CrossTask
Metrics: F1 IoU precision recall
Baselines: CnC GPL-2D UG-I3D GPL STEPS TCC + PCM LAV + PCM Shen et al. Kukleva et al. Elhamifar et al.
Builds on: Sinkhorn distance Cross-view procedure learning (CnC) GPL LAV TCC
Closest papers by meaning
- E-MoFlow: Learning Egomotion and Optical Flow from Event Data via Implicit Regularization (NeurIPS 2025 · similarity 0.84)
- VEDIT: Latent Prediction Architecture For Procedural Video Representation Learning (ICLR 2025 · similarity 0.84)
- OTMatch: Improving Semi-Supervised Learning with Optimal Transport (ICML 2024 · similarity 0.83)
- ShowHowTo: Generating Scene-Conditioned Step-by-Step Visual Instructions (CVPR 2025 · similarity 0.83)
- Emergent Temporal Correspondences from Video Diffusion Transformers (NeurIPS 2025 · similarity 0.83)
- RLGF: Reinforcement Learning with Geometric Feedback for Autonomous Driving Video Generation (NeurIPS 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.