Research map / Multi-agent systems and games / Human-AI and goal-conditioned
AnySkill: Learning Open-Vocabulary Physical Skill for Interactive Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AnySkill, a hierarchical physics-based humanoid controller that composes GAIL-trained atomic latent skills via a high-level PPO policy rewarded by CLIP similarity of rendered images to open-vocabulary text.
Problem
Physics-based humanoid motion generation relying on imitation learning and handcrafted rewards generalizes poorly to novel open-vocabulary instructions and to interactions with objects.
Approach
Use a frozen CLIP model's similarity between rendered agent images and the text as the high-level reward, so a latent-skill policy composes imitation-learned atomic actions for any instruction and object interaction without manual reward engineering.
Key results
- AnySkill (full) achieves success 6.16 and CLIP S 24.18 vs. MotionCLIP 3.16 / 21.16 and AvatarCLIP 4.29 / 21.11 in the user study and CLIP metric (Table 1).
- Removing early termination drops success from 6.16 to 5.05 and CLIP S from 24.18 to 21.89 (Table 1).
- In reward comparisons, AnySkill's CLIP-based reward outperforms VLM-RMs, CLIP-S, AvgPool and velocity reward on all user-study metrics (Table 3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add early termination during high-level policy training (ablation) | AnySkill · Open-vocabulary instructions (5 tasks) | Success: 5.05 -> 6.16 | Helped |
| add text enhancement to instructions (ablation) | AnySkill · Open-vocabulary instructions (5 tasks) | Success: 3.06 -> 6.16 | Helped |
| use the image-based CLIP reward instead of VideoCLIP (ablation) | AnySkill · Open-vocabulary instructions (5 tasks) | Success: 2.37 -> 6.16 | Helped |
| use the image-based CLIP reward instead of VLM-RMs (ablation) | AnySkill · Open-vocabulary instructions (5 tasks) | Success: 3.15 -> 6.16 | Helped |
| use the image-based CLIP reward instead of CLIP-S (ablation) | AnySkill · Open-vocabulary instructions (5 tasks) | Success: 3.80 -> 6.16 | Helped |
| use the image-based CLIP reward instead of AvgPool (ablation) | AnySkill · Open-vocabulary instructions (5 tasks) | Success: 5.09 -> 6.16 | Helped |
| replace image-based CLIP reward with AvgPool (ablation) | AnySkill · Open-vocabulary instructions (5 tasks) | Smoothness: 6.51 -> 6.55 | Helped |
| use AnySkill rather than AvatarCLIP | Open-vocabulary motion generation · Open-vocabulary instructions (5 tasks) | Success: 4.29 -> 6.16 | Helped |
| use AnySkill rather than MotionCLIP | Open-vocabulary motion generation · Open-vocabulary instructions (5 tasks) | Naturalness: 4.93 -> 6.23 | Helped |
Surprising findings
- Using a VideoCLIP-based reward gives the highest physics score among the ablations but low success (2.37), suggesting success and physical plausibility trade off.
Limitations
- Reliance on CLIP and image-based rewards struggles with prolonged actions and visual ambiguity.
- Vague descriptions (e.g., 'do yoga') and extended tasks (e.g., 'walk in a circle') degrade performance.
- Interaction experiments mainly involve a single object.
Experimental setup
Datasets: CMU Graphics Lab Motion Capture Database SFU Motion Capture Database HumanML3D BABEL
Benchmarks: user study on 5 open-vocabulary descriptions interaction motion evaluation reward function comparison
Metrics: success naturalness smoothness physics plausibility CLIP similarity score
Baselines: MotionCLIP AvatarCLIP VLM-RMs CLIP-S VideoCLIP (AvgPool) ASE
Builds on: CALM GAIL PPO CLIP VLM-RMs
Closest papers by meaning
- GROVE: A Generalized Reward for Learning Open-Vocabulary Physical Skill (CVPR 2025 · similarity 0.89)
- Universal Humanoid Motion Representations for Physics-Based Control (ICLR 2024 · similarity 0.87)
- Learning Interactive Real-World Simulators (ICLR 2024 · similarity 0.86)
- OSVI-WM: One-Shot Visual Imitation for Unseen Tasks using World-Model-Guided Trajectory Generation (NeurIPS 2025 · similarity 0.86)
- Reward-free World Models for Online Imitation Learning (ICML 2025 · similarity 0.86)
- Instant Policy: In-Context Imitation Learning via Graph Diffusion (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.