Research map / Reinforcement learning / Visual and representation RL
SUGAR: A Scalable Human-Video-Driven Generalizable Humanoid Loco-Manipulation Learning Framework
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Sugar converts noisy human-object interaction videos into refined physics-based humanoid skills via a privileged RL refiner, then distills them into a hierarchical diffusion command generator plus command tracker for reference-free loco-manipulation on a Unitree G1.
Problem
Learning generalizable whole-body humanoid loco-manipulation from human videos, where video-extracted kinematic HOI priors are noisy (occlusion, contact artifacts, retargeting errors) and reference-tracking policies fail to generalize or transfer to hardware.
Approach
Treat imperfect video-extracted HOI trajectories and VLM-labeled contacts as coarse priors, refine them with a privileged mimic-style RL policy using a progressive state pool for initialization, then distill into a reference-free hierarchical policy (diffusion command generator trained on execution rollouts + command tracker).
Key results
- On training set, Sugar reaches 89.5% SR on Kick Box and 98.8% on Pick Bottle, outperforming ResMimic and HDMI, which fail on Pick Bottle and Stand Bottle (0%).
- On unseen test configurations, Sugar achieves 99.6% on Sit Chair, 99.2% on Pick Bottle, and 86.3% on Stand Bottle; HDMI gets 8.2% on Sit Chair and 17.3% on Kick Box.
- Zero-shot real-world success over 10 trials: 7/10 Kick Box, 6/10 Push Box, 7/10 Carry Box, 9/10 Sit Chair, 9/10 Pick Bottle, 8/10 Stand Bottle.
Surprising findings
- HDMI attains 94.1% SR on Push Box on training data but drops to 54.6% on test, and 0% on Carry Box, Pick Bottle, and Stand Bottle.
- Removing interaction rewards drops Pick Bottle success from 98.8% to 0% on training data.
Limitations
- Extracted priors are coarse, limiting the framework to coarse-grained interaction skills.
- Data utilization efficiency is relatively low.
- State-based policy hinders deployment; no visual or language input.
Experimental setup
Datasets: human videos collected per task (100 train, 30 test per task)
Benchmarks: six loco-manipulation tasks in IsaacSim (Kick Box, Push Box, Carry Box, Sit Chair, Pick Bottle, Stand Bottle) real-world Unitree G1 trials (10 per task)
Metrics: success rate final object position error
Baselines: ResMimic HDMI w/o refiner w/o PSPI (SSI) w/o PSPI (RSI) w/o IR w/o IRE
Builds on: PPO Diffusion Policy DiT ResMimic HDMI FoundationPose
Closest papers by meaning
- Humanoid Horizon: Extending Task Horizon in Whole-Body Loco-Manipulation via Parallel Training, Dynamic Starting, and Reward Gating (NeurIPS 2026 · similarity 0.91)
- Universal Humanoid Motion Representations for Physics-Based Control (ICLR 2024 · similarity 0.89)
- HumanoidArena: Benchmarking Egocentric Hierarchical Whole-body Learning (NeurIPS 2026 · similarity 0.89)
- Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasets (ICLR 2025 · similarity 0.88)
- Adversarial Locomotion and Motion Imitation for Humanoid Policy Learning (NeurIPS 2025 · similarity 0.88)
- DiT4DiT: Jointly Modeling Video Dynamics and Actions for Generalizable Robot Control (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.