Research map / Reinforcement learning / Visual and representation RL
SkillMimic: Learning Basketball Interaction Skills from Demonstrations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
SkillMimic learns diverse basketball human-object interaction skills in simulated humanoids from HOI demonstrations using a single RL policy with a unified imitation reward that includes a contact graph reward, then composes the skills with a high-level controller for scoring.
Problem
Learning diverse physically simulated humanoid-object interaction skills without skill-specific hand-crafted rewards, from HOI demonstration data.
Approach
A unified, skill-agnostic HOI imitation reward (multiplicative body, object, relative motion, regularization and a contact-graph reward on binary contact states) lets one policy conditioned on a skill label learn many basketball skills, and a high-level controller reuses these skills for long-horizon tasks.
Key results
- SkillMimic reaches 86.7% success on pick up and 79.6% on dribble forward versus 19.6% and 68.5% for DeepMimic* and 0.0% and 13.6% for AMP*.
- On four high-level tasks, SkillMimic achieves 93.04% (heading), 79.92% (circling), 93.40% (throwing), and 80.25% (scoring) success, while PPO, ASE and ASE* fail on throwing and scoring.
- Removing the contact graph reward drops BallPlay-V accuracy from 82.4% to 13.6% for the w/o multiplication variant comparison, and the contact error rises markedly without CGR.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SkillMimic rather than the DeepMimic*-style reward | basketball interaction skill learning · BallPlay-M skills: Pick Up, Dribble Forward, Layup, and Shot | success rate: 19.6% -> 86.7%; 68.5% -> 79.6%; 98.9% -> 99.1%; 97.8% -> 97.9% | Helped |
| Use SkillMimic rather than PPO trained from scratch | high-level basketball task learning · Heading | success rate: 0.70% -> 93.04% | Helped |
| Use SkillMimic rather than PPO trained from scratch | high-level basketball task learning · Circling | success rate: 11.14% -> 79.92% | Helped |
| Use SkillMimic rather than PPO trained from scratch | high-level basketball task learning · Throwing | success rate: 0.00% -> 93.40% | Helped |
| Use SkillMimic rather than PPO trained from scratch | high-level basketball task learning · Scoring | success rate: 0.00% -> 80.25% | Helped |
| Add reward multiplication to combine the imitation sub-rewards (ablation) | SkillMimic · GRAB | accuracy: 27.0% -> 95.4% | Helped |
| Add reward multiplication to combine the imitation sub-rewards (ablation) | SkillMimic · BallPlay-V | accuracy: 7.5% -> 82.4% | Helped |
| Add Contact Graph Reward (ablation) | SkillMimic · GRAB | accuracy: 38.6% -> 95.4% | Helped |
| Add Contact Graph Reward (ablation) | SkillMimic · BallPlay-V | accuracy: 13.6% -> 82.4% | Helped |
| Increase pickup demonstration data from 1 to 10 clips (ablation) | SkillMimic pickup policy · Pickup generalization test | success rate: 0.5% -> 12.2% | Helped |
| Increase pickup demonstration data from 10 to 40 clips (ablation) | SkillMimic pickup policy · Pickup generalization test | success rate: 12.2% -> 76.5% | Helped |
| Increase pickup demonstration data from 40 to 131 clips (ablation) | SkillMimic pickup policy · Pickup generalization test | success rate: 76.5% -> 85.6% | Helped |
Surprising findings
- Adversarial (AMP-style) imitation rewards perform very poorly on HOI imitation, reaching near 0% success on several skills.
- Pickup generalization improves with data scale, from 0.5% success with 1 clip to 85.6% with 131 clips.
Limitations
- Generalization across object geometries and multiple simultaneous objects is not addressed.
- Relies on privileged object information rather than general perception.
- Not yet demonstrated on real robots.
Experimental setup
Datasets: BallPlay-M BallPlay-V GRAB
Benchmarks: BallPlay-M BallPlay-V GRAB
Metrics: accuracy MPJPE body MPJPE object contact error success rate
Baselines: DeepMimic* AMP* PPO ASE ASE*
Builds on: DeepMimic AMP ASE PPO
Closest papers by meaning
- BASKET: A Large-Scale Video Dataset for Fine-Grained Skill Estimation (CVPR 2025 · similarity 0.85)
- Goal-Oriented Skill Abstraction for Offline Multi-Task Reinforcement Learning (ICML 2025 · similarity 0.85)
- InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactions (CVPR 2025 · similarity 0.84)
- ExpertAF: Expert Actionable Feedback from Video (CVPR 2025 · similarity 0.84)
- The CoachAI Badminton Environment: A Novel Reinforcement Learning Environment with Realistic Opponents (Student Abstract) (AAAI 2024 · similarity 0.84)
- Dynamic Contrastive Skill Learning with State-Transition Based Skill Clustering and Dynamic Length Adjustment (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.