Research map / Vision-language and multimodal / Cross-modal learning
Humanoid Locomotion as Next Token Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Casts humanoid locomotion as autoregressive next-token prediction over sensorimotor trajectories with a causal transformer, handling missing modalities via mask tokens, and deploys a Digit humanoid zero-shot in San Francisco.
Problem
Learning real-world humanoid locomotion from heterogeneous sensorimotor data, including trajectories with missing action or sensory modalities such as human videos, where standard RL or behavior cloning cannot easily use such data.
Approach
Model the joint distribution of complete sensorimotor trajectories with a causal transformer trained by autoregressive regression, using modality-aligned prediction and mask tokens so that trajectories without actions (MoCap, YouTube, model-based controller observations) can be used for training.
Key results
- Tracking error of the autoregressive policy is lower than the RL baseline at all turning speeds and across N=245 commands in simulation.
- Prediction error correlates with tracking error with Pearson r=0.87 across 14 models.
- Policy deployed zero-shot on a full-sized Digit humanoid across San Francisco over one week and generalizes to backward walking.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use separate state and action tokens instead of concatenating them (ablation) | Humanoid locomotion transformer · Command set | tracking error: 0.310 -> 0.299 | Helped |
| concatenate state and action into one token instead of using separate tokens (ablation) | Humanoid locomotion transformer · Validation test set | prediction error: 0.98 -> 0.88 | Helped |
| align the predicted modality with the input modality (ablation) | Humanoid locomotion transformer · Command set | tracking error: 0.338 -> 0.299 | Helped |
| align the predicted modality with the input modality (ablation) | Humanoid locomotion transformer · Validation test set | prediction error: 1.05 -> 0.98 | Helped |
| use staged rather than joint training on complete and incomplete trajectories (ablation) | Humanoid locomotion transformer · Command set | tracking error: 0.310 -> 0.311 | No clear effect |
| predict both states and actions instead of actions only (ablation) | Humanoid locomotion transformer · Command set | tracking error: 0.335 -> 0.305 | Helped |
Surprising findings
- Model trained on trajectories from the RL policy outperforms that same RL policy on tracking error.
- Backward walking emerges zero-shot despite no backward commands in action-labeled training data.
Limitations
- Tracking error reported mostly in simulation; real-world evaluation is qualitative deployment rather than quantitative metrics.
- YouTube-derived trajectories are noisy due to reconstruction and retargeting.
- Regression on raw token values rather than quantized or vector-quantized tokens was chosen for simplicity without comparison.
Experimental setup
Datasets: Neural network policy trajectories (simulated, Agility Robotics simulator) Model-based controller trajectories (Agility Robotics) KIT motion capture via AMASS YouTube human videos reconstructed with PHALP
Benchmarks: MuJoCo simulated velocity command tracking real-world Digit deployment in San Francisco
Metrics: position tracking error next token prediction error phase portrait of knee joint
Baselines: RL tracking controller (neural network policy trained with reinforcement learnin Agility Robotics model-based controller
Builds on: GPT Isaac Gym RL locomotion policy AMASS PHALP SMPL
Closest papers by meaning
- H-GAP: Humanoid Control with a Generalist Planner (ICLR 2024 · similarity 0.87)
- Universal Humanoid Motion Representations for Physics-Based Control (ICLR 2024 · similarity 0.87)
- SUGAR: A Scalable Human-Video-Driven Generalizable Humanoid Loco-Manipulation Learning Framework (NeurIPS 2026 · similarity 0.87)
- Hierarchical World Models as Visual Whole-Body Humanoid Controllers (ICLR 2025 · similarity 0.87)
- Humanoid Horizon: Extending Task Horizon in Whole-Body Loco-Manipulation via Parallel Training, Dynamic Starting, and Reward Gating (NeurIPS 2026 · similarity 0.87)
- Self-Improving Embodied Foundation Models (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.