Research map / Reinforcement learning / Imitation learning
Robot Policy Learning with Temporal Optimal Transport Reward
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TemporalOT, an optimal-transport proxy reward that adds temporal order via context-embedding cost matrices and a temporal band mask, learning Meta-world robot policies from only two expert videos.
Problem
OT-based proxy rewards for imitation from a few expert videos are invariant to temporal order, adding reward noise and allowing convergence to undesired behaviors.
Approach
Replace pairwise cosine cost with a group-wise (context-window) cosine cost and apply a diagonal band temporal mask inside the entropic OT (Sinkhorn) plan so the reward focuses on temporally nearby expert frames.
Key results
- TemporalOT achieves 61.1% average success across nine Meta-world tasks versus 35.9% for ADS and 20.8% for TaskReward.
- TemporalOT beats ADS on 8 of 9 tasks, e.g., Stick-push 97.6 vs 20.0 and Door-lock 33.4 vs 3.2.
- TemporalOT is weaker than ADS on Push (8.4 vs 17.2).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use TemporalOT instead of the strongest baseline | robot policy learning with two expert video demonstrations · Meta-world benchmark tasks | success rate: 61.1 vs 35.9 average; TemporalOT outperforms ADS in 8 out of 9 tasks | Mixed |
| remove the temporal mask (ablation) | TemporalOT · Meta-world benchmark tasks | success rate | Hurt |
| add a temporal mask (ablation) | TemporalOT · Door-open | success rate | Helped |
| remove the context cost matrix (ablation) | TemporalOT · Meta-world benchmark tasks | success rate | Hurt |
| add the context cost matrix (ablation) | TemporalOT · Stick-push | success rate | Helped |
| pretrain with behavior cloning before TemporalOT fine-tuning (ablation) | TemporalOT · Door-open | success rate: 78.4 -> 82.0 | Helped |
| use a 2x-speed expert demonstration (ablation) | TemporalOT · Basketball | success rate: 94.4 (4.7) -> 91.8 (8.9) | Hurt |
| use faster expert demonstrations (ablation) | TemporalOT · Basketball, Button-press, and Door-open | success rate: Basketball: 94.4 (4.7) -> 77.2 (8.4) -> 43.8 (29.5); Button-press: 89.0 (3.8) -> 65.6 (5.4) -> 57.8 (7.4); Door-open: 78 | Hurt |
| use a dynamic mask instead of a fixed rule-based temporal mask (ablation) | TemporalOT · Meta-world benchmark tasks | success rate | Helped |
| replace ResNet50 with ResNet18 (ablation) | TemporalOT visual encoder · Meta-world benchmark tasks | success rate | Hurt |
| increase the context length beyond a medium value (ablation) | TemporalOT · Meta-world benchmark tasks | success rate | Hurt |
| increase the number of expert demonstrations (ablation) | TemporalOT · Meta-world benchmark tasks | success rate | Helped |
Surprising findings
- Oracle task-reward RL (TaskReward) fails on most tasks (average 20.8%) because it rarely collects a first successful trajectory under sparse reward.
Limitations
- Depends on quality of expert demonstrations; sub-optimal or biased demos are inherited.
- Performance relies on the pretrained visual encoder; fails when it misses key details (e.g., low-contrast block in Hand-insert).
- Computation cost scales with the number of expert demonstrations.
Experimental setup
Datasets: Meta-world expert video demonstrations (2 per task)
Benchmarks: Meta-world (9 tasks)
Metrics: success rate
Baselines: TaskReward (DrQ-v2) BC GAIfO OT (OTR online) with gamma 0.99 OT with gamma 0.9 ADS
Builds on: Optimal Transport Reward Labeling (OTR) Automatic Discount Scheduling (ADS) DrQ-v2 GAIfO Sinkhorn algorithm
Closest papers by meaning
- Video-Based Optimal Transport for Feedback-Efficient Offline Preference-Based Reinforcement Learning (ICML 2026 · similarity 0.87)
- Optimal Transport for Offline Imitation Learning (ICLR 2023 · similarity 0.87)
- Score-Based Diffusion Policy Compatible with Reinforcement Learning via Optimal Transport (ICML 2025 · similarity 0.86)
- PEARL: Zero-shot Cross-task Preference Alignment and Robust Reward Learning for Robotic Manipulation (ICML 2024 · similarity 0.85)
- Going Beyond Heuristics by Imposing Policy Improvement as a Constraint (NeurIPS 2024 · similarity 0.85)
- Efficient Test-Time Adaptation For Robot Policies (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.