Research map / Reinforcement learning / Visual and representation RL

Efficient Language-instructed Skill Acquisition via Reward-Policy Co-Evolution

AAAI 2025 · Reinforcement learning › Visual and representation RL · approach: Skill learning and diffusion policies · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ROSKA, a reward-policy co-evolution framework where LLM-generated reward candidates are evaluated by fine-tuning the previous best policy (fused with random parameters at a ratio chosen by Short-Cut Bayesian Optimization), using a dynamic candidate population.

Problem

Eureka-style LLM reward design re-trains a policy from scratch for every candidate reward, which is sample-inefficient and forces one universal reward to work across the whole policy improvement process, for high-dimensional robotic control tasks.

Approach

Co-evolve rewards and policies: LLM reward candidates are scored by warm-starting from the previous best policy, blended with random parameters via a fusion ratio chosen by early-stopped Bayesian optimization (SC-BO), and the candidate population size grows dynamically until an improvement over the previous best is found.

Key results

• ROSKA reaches MTS 24.34±2.84 on ShadowHand vs 9.56±2.17 for Eureka and 6.00±1.02 for human reward, with about 0.89x Eureka's training samples.
• Averaged over tasks, ROSKA reports a 95.3% normalized improvement over Eureka on the HNS metric.
• ROSKA-U (uniform search) reaches comparable scores (e.g., 26.80 on AllegroHand) but uses about 2.2x Eureka's training samples.

Measured effects reported in the paper

Surprising findings

• Eureka's reward-only evaluation sometimes yields later rounds that underperform earlier ones, which the authors attribute to retraining from scratch each round.
• Fixed fusion ratios of 0% (fully random re-initialization) underperform the inherited policies on Ant and ShadowHand, yet 100% inheritance also underperforms the adaptive SC-BO choice on Humanoid.

Limitations

• LLM-generated reward functions are unstable across queries, especially on Humanoid where first-round candidates may all be unworkable.
• Evaluated only on six IsaacGym tasks with dense or sparse task fitness; no real-robot experiments.
• Gains depend on the LLM producing good early rewards; the method is sensitive to initial reward quality.

Experimental setup

Datasets: IsaacGym simulated environments (Ant, Humanoid, ShadowHand, AllegroHand, FrankaCabinet, ShadowHandUpsideDown)

Benchmarks: IsaacGym Ant IsaacGym Humanoid IsaacGym ShadowHand IsaacGym AllegroHand IsaacGym FrankaCabinet IsaacGym ShadowHandUpsideDown

Metrics: Max Training Success (MTS) Human Normalized Score (HNS) Total Training Samples (TTS)

Baselines: Sparse reward Human reward Eureka ROSKA-U (uniform fusion-ratio search) ROSKA-0% ROSKA-50% ROSKA-100%

Builds on: Eureka PPO Expected Improvement Bayesian optimization

Closest papers by meaning

• Eureka: Human-Level Reward Design via Coding Large Language Models (ICLR 2024 · similarity 0.91)
• Text2Reward: Reward Shaping with Language Models for Reinforcement Learning (ICLR 2024 · similarity 0.91)
• Learning Reward for Robot Skills Using Large Language Models via Self-Alignment (ICML 2024 · similarity 0.91)
• R*: Efficient Reward Design via Reward Structure Evolution and Parameter Alignment Optimization with Large Language Models (ICML 2025 · similarity 0.91)
• REvolve: Reward Evolution with Large Language Models using Human Feedback (ICLR 2025 · similarity 0.90)
• Progress Reward Model for Reinforcement Learning via Large Language Models (NeurIPS 2025 · similarity 0.90)