Research map / Vision-language and multimodal / Robot action and navigation
Provable Ordering and Continuity in Vision-Language Pretraining for Generalizable Embodied Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AcTOL, a vision-language pretraining method on human action videos that combines a temporal ordering contrastive loss with a Brownian bridge continuity constraint, showing gains in language-conditioned imitation learning on Franka Kitchen, Metaworld, and a real Unitree D1 arm.
Problem
Goal-reaching time contrastive pretraining on noisy human action videos aligns language with late frames, producing erroneous vision-language associations when actions end early or include irrelevant moments, which hurts downstream robot imitation learning.
Approach
Replace goal-reaching alignment with a vision-language ordering loss that makes semantic similarity differences track temporal distance between frames, plus a local Brownian bridge loss that enforces smooth intermediate frame trajectories; theoretical bounds on ordering, continuity, and robustness to language perturbations are provided.
Key results
- On Franka Kitchen with 5 demos AcTOL reaches 42.6% success vs 25.3% for DecisionNCE (strongest baseline), and with 25 demos 64.6% vs 50.9%.
- On Metaworld with 25 demos AcTOL reaches 84.9% vs 75.8% for DecisionNCE.
- Under linguistic perturbations, LIV and DecisionNCE drop by 11.9% and 2.7% on average while AcTOL drops by 0.9%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use AcTOL instead of the strongest baseline | Franka Kitchen LCBC · Franka Kitchen, 5 demos | success rate: 25.33 ± 1.30 -> 42.60 ± 0.53 | Helped |
| use AcTOL instead of the strongest baseline | Franka Kitchen LCBC · Franka Kitchen, 15 demos | success rate: 43.20 ± 2.25 -> 61.80 ± 2.54 | Helped |
| use AcTOL instead of the strongest baseline | Franka Kitchen LCBC · Franka Kitchen, 25 demos | success rate: 51.93 ± 0.95 -> 64.60 ± 0.57 | Helped |
| use AcTOL instead of the strongest baseline | Metaworld LCBC · Metaworld, 5 demos | success rate: 46.95 ± 2.07 -> 53.81 ± 3.89 | Helped |
| use AcTOL instead of the strongest baseline | Metaworld LCBC · Metaworld, 15 demos | success rate: 64.33 ± 3.63 -> 74.13 ± 1.59 | Helped |
| use AcTOL instead of the strongest baseline | Metaworld LCBC · Metaworld, 25 demos | success rate: 69.75 ± 3.90 -> 81.13 ± 1.59 | Helped |
| add the Brownian Bridge loss (ablation) | AcTOL · Franka Kitchen LCBC, 5 demos | success rate: 32.80 ± 1.23 -> 42.60 ± 0.53 | Helped |
| add the Brownian Bridge loss (ablation) | AcTOL · Franka Kitchen LCBC, 15 demos | success rate: 54.20 ± 0.85 -> 61.80 ± 2.54 | Helped |
| add the Brownian Bridge loss (ablation) | AcTOL · Franka Kitchen LCBC, 25 demos | success rate: 60.80 ± 0.87 -> 64.60 ± 0.57 | Helped |
| add the Brownian Bridge loss (ablation) | AcTOL · Metaworld LCBC, 5 demos | success rate: 50.29 ± 4.05 -> 53.81 ± 3.89 | Helped |
| add the Brownian Bridge loss (ablation) | AcTOL · Metaworld LCBC, 15 demos | success rate: 70.83 ± 4.21 -> 74.13 ± 1.59 | Helped |
| add the Brownian Bridge loss (ablation) | AcTOL · Metaworld LCBC, 25 demos | success rate: 73.33 ± 2.83 -> 81.13 ± 1.59 | Helped |
Surprising findings
- Fine-tuning the AcTOL encoders on 25 robot demonstrations raises Franka Kitchen success from 61.8% to 86.4%.
- Removing the Brownian bridge loss still leaves AcTOL above all baselines, with the BB term contributing a further gain (e.g., 54.2% to 61.8% at 15 demos).
Limitations
- May not fit tasks with ambiguous, repetitive, or cyclic behaviors, where the assumption of coherent temporal progression breaks down.
- Real-robot evaluation uses only three tasks and 10 evaluation runs per task.
- Theoretical results are stated under Lipschitz and idealized assumptions, and the paper gives no empirical verification of the bounds.
Experimental setup
Datasets: EPIC-KITCHENS-100 Franka Kitchen demonstrations Metaworld demonstrations Unitree D1 real robot demonstrations
Benchmarks: Franka Kitchen Metaworld Unitree D1 real robot tasks
Metrics: success rate relative performance gain
Baselines: CLIP R3M LIV DecisionNCE AcTOL w/o BB
Builds on: CLIP LIV DecisionNCE R3M
Closest papers by meaning
- Continuous Vision-Language-Action Co-Learning with Semantic-Physical Alignment for Behavioral Cloning (AAAI 2026 · similarity 0.90)
- LangForce: Bayesian Decomposition of Vision Language Action Models via Latent Action Queries (ICML 2026 · similarity 0.89)
- UniJEPA: Enhancing Robot Policy via Unified Continuous and Discrete Representation Learning (ICML 2026 · similarity 0.88)
- Latent Action Pretraining from Videos (ICLR 2025 · similarity 0.88)
- Beyond World-Frame Action Heads: Motion-Centric Action Frames for Vision-Language-Action Models (NeurIPS 2026 · similarity 0.87)
- Beyond Data Scaling: Representation-Centric Pre-training for Vision-Language-Action Models (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.