Research map / Reinforcement learning / Visual and representation RL
Hierarchical World Models as Visual Whole-Body Humanoid Controllers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Puppeteer trains a single 5M-parameter TD-MPC2 tracking world model on MoCap clips, then a high-level TD-MPC2 puppeteer that outputs end-effector commands from visual input, yielding performant and more natural visual whole-body humanoid control.
Problem
Visual whole-body control of a high-dimensional 56-DoF simulated humanoid with RL is hard; prior hierarchical methods rely on many per-clip skill policies or privileged state, and rewards alone do not yield natural motion.
Approach
A two-level hierarchical world model: a single pretrained proprioceptive tracking world model (trained on MoCap via offline+online data) is frozen and reused, while a high-level visual TD-MPC2 puppeteer outputs end-effector position commands for it, plus a termination head with soft truncation during planning.
Key results
- Puppeteer matches TD-MPC2 episode return on most of the 8 tasks (except stairs, where TD-MPC2 is higher), while SAC and DreamerV3 achieve no meaningful performance within 3M steps.
- In a user study (n=51), 97.8% of preference responses rated Puppeteer motions as more natural, 1.8% equal, and 0.4% TD-MPC2.
- The single tracking world model achieves 88.3% tracking success rate across 836 clips, versus 6.2% for offline-only training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the hierarchical Puppeteer controller instead of end-to-end TD-MPC2 | visual whole-body control · Humanoid motion pairs, aggregate user study | preference: 0.4% -> 97.8% | Helped |
| Use Puppeteer instead of TD-MPC2 | Gaps · Gaps | episode length at 1M steps: 66.9 ± 9.8 -> 115.9 ± 5.2 | Helped |
| Use Puppeteer instead of TD-MPC2 | visual whole-body control · Gaps | torso height (cm): 85.9 ± 4.7 -> 96.0 ± 0.2 | Helped |
| Use Puppeteer instead of TD-MPC2 | visual whole-body control · Gaps | episode length: 181.6 ± 28.1 -> 159.3 ± 5.9 | Hurt |
| Add online interaction data to offline pretraining data (ablation) | Puppeteer tracking model · MoCap tracking | tracking quality: null | Helped |
| Add diverse MoCap clips, using all 836 clips (ablation) | Puppeteer tracking model · MoCap tracking | tracking quality: null | Helped |
| Add planning instead of using a learned model-free policy (ablation) | both levels of Puppeteer · downstream tasks | normalized score: null | Helped |
| Add high-level pretraining on the corridor task before task-specific finetuning (ablation) | Puppeteer high-level agent · visual control tasks | normalized score: null | Helped |
| Generalize to gap lengths up to 3× the training data without additional training (ablation) | Puppeteer · Gaps | normalized performance: 3× | Helped |
| Use Puppeteer instead of TD-MPC2 | visual whole-body control · Stairs | episode return | Hurt |
| Use SAC or DreamerV3 instead of Puppeteer | whole-body humanoid control · benchmark tasks | episode return | Hurt |
Surprising findings
- Vanilla SAC and DreamerV3 failed to achieve non-trivial performance on the high-dimensional task suite despite tuning.
- TD-MPC2 reaches higher return on stairs but by rolling up stairs rather than walking, suggesting reward hacking.
Limitations
- Tasks primarily evaluate visio-locomotive capabilities, limiting breadth of the benchmark.
- Only the low-level tracking agent is pretrained in most experiments; high-level pretraining gave a limited experiment.
- Inference is slower (88.2 ms/step) than TD-MPC2 and SAC due to planning.
Experimental setup
Datasets: MoCapAct CMU MoCap
Benchmarks: Puppeteer task suite (8 tasks: corridor, hurdles, walls, gaps, stairs, stand, walk, run)
Metrics: episode return normalized score tracking success rate tracking error CoMic score episode length torso height human preference rate
Baselines: SAC DreamerV3 TD-MPC2 hierarchical SAC hierarchical DreamerV3
Builds on: TD-MPC2 MoCapAct DreamerV3 CoMic
Closest papers by meaning
- HumanoidArena: Benchmarking Egocentric Hierarchical Whole-body Learning (NeurIPS 2026 · similarity 0.91)
- Humanoid Horizon: Extending Task Horizon in Whole-Body Loco-Manipulation via Parallel Training, Dynamic Starting, and Reward Gating (NeurIPS 2026 · similarity 0.89)
- Universal Humanoid Motion Representations for Physics-Based Control (ICLR 2024 · similarity 0.89)
- Adversarial Locomotion and Motion Imitation for Humanoid Policy Learning (NeurIPS 2025 · similarity 0.89)
- H-GAP: Humanoid Control with a Generalist Planner (ICLR 2024 · similarity 0.88)
- SUGAR: A Scalable Human-Video-Driven Generalizable Humanoid Loco-Manipulation Learning Framework (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.