Research map / Reinforcement learning / Visual and representation RL
InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
InterMimic trains per-subject physics-based teacher policies to imitate and refine imperfect human-object interaction MoCap data, then distills them into a transformer student policy with RL fine-tuning for scalable whole-body HOI control.
Problem
Physics-based imitation of whole-body human-object interactions from large, imperfect motion capture data is hard due to contact artifacts, object geometry variability, retargeting issues, and poor hand detail.
Approach
A curriculum of "perfect first, then scale": MLP teacher policies per subject correct MoCap errors and retarget to a canonical body via contact-guided rewards and physical state initialization; a transformer student is trained by combining policy distillation (DAgger to PPO schedule) with reference distillation from teachers, then RL fine-tuned to surpass demonstrations.
Key results
- On OMOMO test, the full student (transformer, reference + policy distillation, PPO) reaches 88.8%/98.1% success on train/test-style splits with ~176 s duration and 5.9 cm human error (Table 2).
- On BEHAVE yogamat single-object imitation, the teacher achieves 42.6 s duration vs 12.2 s for SkillMimic, with lower object error (9.2 vs 13.4 cm).
- Transformer student outperforms MLP student, and PPO fine-tuning after DAgger distillation improves over pure policy distillation on all metrics.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add interaction early termination (ablation) | InterMimic teacher · BEHAVE yogamat | Duration (s): 36.1 -> 42.6 | Helped |
| Add physical state initialization (ablation) | InterMimic teacher · BEHAVE yogamat | Duration (s): 36.1 -> 42.6 | Helped |
| Use teacher-refined references instead of imperfect MoCap references (ablation) | InterMimic student · OMOMO-Test | Success Rate (%): 54.3 -> 91.6 | Helped |
| Add PPO fine-tuning to policy distillation (ablation) | InterMimic student · OMOMO-Test | Success Rate (%): 54.3 -> 91.6 | Helped |
| Use a Transformer instead of an MLP policy (ablation) | InterMimic student · OMOMO-Test | Success Rate (%): 91.6 -> 98.1 | Helped |
Surprising findings
- Training a policy from scratch with PPO or relying solely on DAgger distillation fails to reach optimal performance, indicating RL fine-tuning is needed to resolve teacher conflicts.
Limitations
- Some MoCap data discarded due to severe errors that could not be corrected.
- Excludes soft-body interactions because the simulator (Isaac Gym) lacks full support.
Experimental setup
Datasets: OMOMO BEHAVE HODome IMHD HIMO InterAct
Benchmarks: OMOMO test BEHAVE HOI-Diff generations InterDiff generations
Metrics: success rate duration human tracking error object tracking error
Baselines: PhysHOI SkillMimic PPO from scratch DAgger policy distillation
Builds on: PPO DAgger Reference State Initialization PhysHOI AMP
Closest papers by meaning
- InterControl: Zero-shot Human Interaction Generation by Controlling Every Joint (NeurIPS 2024 · similarity 0.90)
- InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generation (CVPR 2025 · similarity 0.90)
- InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interaction (NeurIPS 2024 · similarity 0.89)
- CG-HOI: Contact-Guided 3D Human-Object Interaction Generation (CVPR 2024 · similarity 0.88)
- DiffGrasp: Whole-Body Grasping Synthesis Guided by Object Motion Using a Diffusion Model (AAAI 2025 · similarity 0.88)
- CoDA: Coordinated Diffusion Noise Optimization for Whole-Body Manipulation of Articulated Objects (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.