Research map / 3D vision / 3D generation
TokenHSI: Unified Synthesis of Physical Human-Scene Interactions through Task Tokenization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
TokenHSI trains a single transformer policy with a shared proprioception token and per-task tokens (masked attention) for physics-based human-scene interaction skills, and adapts it to new composite, shape-variation, terrain and long-horizon tasks by training only lightweight tokenizers and adapters.
Problem
Physics-based human-scene interaction (HSI) controllers are usually specialized for single tasks, and unified controllers struggle with dynamic manipulation skills, multi-skill composition, and efficient adaptation to new scenes.
Approach
Tokenize the humanoid proprioception as a separate shared token and each task observation as its own task token, using a one-hot-label attention mask in a transformer encoder so one policy learns multiple HSI skills; adapt to new tasks by adding task tokenizers and zero-initialized action-head adapters while freezing the rest.
Key results
- Unified policy matches or exceeds per-task specialists on four foundational skills, e.g., Carry success 92.2% vs 83.1% for the specialist.
- On Climb + Carry composition, TokenHSI reaches 99.2% success vs 68.3% for CML and 26.8% for Scratch.
- Object shape variation: TokenHSI achieves 88.8% on Chair vs 84.5% for AdaptNet, with better learning efficiency than AdaptNet.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use a unified TokenHSI policy instead of specialist policies | foundational HSI skills · Follow, Sit, Climb, and Carry | success rate: Follow: 98.7±0.5 -> 99.7±0.0; Sit: 98.2±2.0 -> 99.6±0.2; Climb: 99.7±0.1 -> 99.8±0.1; Carry: 83.1±5.0 -> 92.2±6.7 | Helped |
| use a unified TokenHSI policy instead of specialist policies | foundational HSI skills · Follow, Sit, Climb, and Carry | error (cm): Follow: 6.5±0.0 -> 9.3±0.1; Sit: 5.6±0.0 -> 5.6±0.2; Climb: 2.4±0.2 -> 2.7±0.3; Carry: 5.1±0.2 -> 4.2±0.6 | Mixed |
| add the shared proprioception tokenizer Tprop (ablation) | TokenHSI foundational skills · Follow, Sit, Climb, and Carry | success rate: Follow: 99.3±0.3 -> 99.7±0.0; Sit: 98.7±0.4 -> 99.6±0.2; Climb: 99.5±0.2 -> 99.8±0.1; Carry: 90.9±3.3 -> 92.2±6.7 | Helped |
| compose the climbing skill with carrying | TokenHSI · Climb + Carry | success rate: 68.3±21.9 -> 99.2±0.1 | Helped |
| compose the sitting skill with carrying | TokenHSI · Sit + Carry | error (cm): 6.2±0.2 -> 5.5±0.0 | Helped |
| compose the follow skill with carrying | TokenHSI · Follow + Carry | success rate: 99.1±0.2 -> 99.1±0.4 | No clear effect |
| add the shared proprioception tokenizer Tprop (ablation) | TokenHSI skill composition · Climb + Carry | success rate: 81.9±9.0 -> 99.2±0.1 | Helped |
| adapt TokenHSI to carrying chairs rather than boxes | TokenHSI · Chair carrying | success rate: 87.5±0.6 -> 88.8±3.1 | Helped |
| adapt TokenHSI to carrying tables rather than boxes | TokenHSI · Table carrying | success rate: 83.4±1.6 -> 83.6±1.6 | No clear effect |
| adapt the follow skill to uneven terrain | TokenHSI · Follow on terrain | success rate: 92.4±0.5 -> 96.0±0.4 | Helped |
| adapt the carry skill to uneven terrain | TokenHSI · Carry on terrain | success rate: 63.4±0.6 -> 74.0±2.5 | Helped |
| add adapter layers to the action head (ablation) | TokenHSI terrain adaptation · Follow on terrain | success rate: 63.0±1.1 -> 96.0±0.4 | Helped |
Surprising findings
- Removing the shared proprioception tokenizer hurts composition adaptation most on the hardest task (Climb + Carry drops to 81.9% from 99.2%).
Limitations
- Learning skills requires tedious reward engineering via trial and error.
- Long-horizon task completion remains non-autonomous and relies on a finite state machine for task switching.
- Evaluated only in simulation on a limited set of four foundational skills.
Experimental setup
Datasets: procedurally generated paths object datasets for sitting, climbing, carrying, chairs and tables
Benchmarks: path following tasks (simulated) sit, climb, carry tasks (simulated) skill composition tasks object shape variation (chair, table) terrain shape variation (stairs, obstacles) long-horizon task
Metrics: success rate (%) error (cm) learning curve iterations
Baselines: AMP specialists Scratch CML CML (dual) Finetune AdaptNet Ours (w/o Tprop)
Builds on: AMP PPO AdaptNet CML UniHSI
Closest papers by meaning
- Unified Human-Scene Interaction via Prompted Chain-of-Contacts (ICLR 2024 · similarity 0.88)
- HSI-GPT: A General-Purpose Large Scene-Motion-Language Model for Human Scene Interaction (CVPR 2025 · similarity 0.87)
- PhyScene: Physically Interactable 3D Scene Synthesis for Embodied AI (CVPR 2024 · similarity 0.86)
- OpenHOI: Open-World Hand-Object Interaction Synthesis with Multimodal Large Language Model (NeurIPS 2025 · similarity 0.86)
- Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models (NeurIPS 2026 · similarity 0.85)
- MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D World (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.