Research map / Large language models / Agents, code and math
GROVE: A Generalized Reward for Learning Open-Vocabulary Physical Skill
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GROVE, a reward framework that combines LLM-generated physical constraint rewards with CLIP-based VLM semantic rewards (via a Pose2CLIP pose-to-CLIP mapper) and regenerates LLM rewards when VLM fitness declines, to learn open-vocabulary humanoid and standard RL skills without manual reward design.
Problem
Learning open-vocabulary physical skills for simulated agents is hard: hand-designed rewards do not scale across tasks, and demonstration-based imitation does not generalize beyond its training distribution.
Approach
Sum an LLM-generated constraint reward (Eureka-style, GPT-o1-preview) with a CLIP-based semantic reward computed through Pose2CLIP, a lightweight MLP that maps agent joint poses directly to CLIP image features without rendering; the VLM score doubles as a fitness signal that triggers LLM reward regeneration when it stalls or declines.
Key results
- GROVE reaches task completion 7.924 vs 6.108 for AnySkill (VLM-only), a 25.7% gain, with motion naturalness 6.793 and training time 7 min vs 59 min.
- On the humanoid bowing task GROVE scores 6.276 completion in 47 min versus VLM-RM's 3.483 with improved textures at 411 min (about 11.4% of training time).
- Reward distance on standard RL tasks: GROVE beats direct expert-reward optimization on 3 of 4 tasks (e.g., Ant 92.45 vs 103.81) but loses on ANYmal jumping (60.84 vs 57.29).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| combine LLM reward with the VLM reward (ablation) | VLM-only humanoid skill learning with direct CLIP features · Open-vocabulary humanoid skills | task completion (C): 6.108 -> 6.954 | Helped |
| replace direct CLIP features with Pose2CLIP features (ablation) | VLM-only humanoid skill learning · Open-vocabulary humanoid skills | task completion (C): 6.108 -> 6.785 | Helped |
| replace direct CLIP features with Pose2CLIP features (ablation) | VLM-only humanoid skill learning · Open-vocabulary humanoid skills | training time: 59min -> 16min | Helped |
| add LLM reward to Pose2CLIP-based VLM reward (ablation) | Pose2CLIP-only humanoid skill learning · Open-vocabulary humanoid skills | task completion (C): 6.785 -> 7.924 | Helped |
| add RDP-based reward resampling (ablation) | Pose2CLIP + LLM humanoid skill learning · Open-vocabulary humanoid skills | task completion (C): 7.269 -> 7.924 | Helped |
| add RDP-based reward resampling (ablation) | LLM-only humanoid skill learning · Open-vocabulary humanoid skills | task completion (C): 6.622 -> 6.650 | No clear effect |
| combine LLM and VLM rewards (ablation) | humanoid skill learning with Pose2CLIP · Open-vocabulary humanoid skills | task completion (C): 6.785 -> 7.924 | Helped |
| use GROVE instead of direct reward optimization | Ant locomotion policy · Ant running forward | reward distance (RD): 103.81 -> 92.45 | Helped |
| use GROVE instead of direct reward optimization | Cartpole policy · Cartpole keep upright | reward distance (RD): 20.72 -> 20.04 | Helped |
| use GROVE instead of direct reward optimization | ANYmal policy · ANYmal jump up | reward distance (RD): 57.29 -> 60.84 | Hurt |
| use GROVE instead of direct reward optimization | ANYmal policy · ANYmal walk sideways | reward distance (RD): 39.71 -> 37.51 | Helped |
Surprising findings
- Direct optimization of the evaluation reward outperformed GROVE on the ANYmal jumping task, which the authors attribute to the task's multiple success criteria.
- Pose2CLIP features alone (without LLM reward) improved CLIP similarity and completion over raw simulation-rendered CLIP features in the VLM-only setting, suggesting the domain gap matters.
Limitations
- Evaluation relies on 30 human raters and a small set of five open-vocabulary instructions plus a few RL tasks.
- Humanoid experiments depend on a pre-trained CALM low-level controller, and the LLM reward is limited to what GPT-o1-preview generates.
- The VLM fitness threshold and regeneration rule (8 consecutive drops, 0.1 threshold) are hand-chosen heuristics.
Experimental setup
Datasets: AMASS Motion-X policy rollouts (Pose2CLIP augmentation)
Benchmarks: AvatarCLIP/AnySkill open-vocabulary humanoid instructions Ant Cartpole ANYmal Humanoid bowing
Metrics: task completion score motion naturalness smoothness physical realism CLIP similarity reward distance training time
Baselines: AvatarCLIP TMR MoMask MotionGPT AnySkill VLM-RM Eureka-style LLM reward
Builds on: Eureka VLM-RM CALM AnySkill PPO
Closest papers by meaning
- AnySkill: Learning Open-Vocabulary Physical Skill for Interactive Agents (CVPR 2024 · similarity 0.89)
- Eureka: Human-Level Reward Design via Coding Large Language Models (ICLR 2024 · similarity 0.89)
- GoalLadder: Incremental Goal Discovery with Vision-Language Models (NeurIPS 2025 · similarity 0.88)
- Language Guided Skill Discovery (ICLR 2025 · similarity 0.88)
- Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning (ICLR 2024 · similarity 0.88)
- Text2Reward: Reward Shaping with Language Models for Reinforcement Learning (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.