Research map / Large language models / Alignment and preferences

R*: Efficient Reward Design via Reward Structure Evolution and Parameter Alignment Optimization with Large Language Models

ICML 2025 · Large language models › Alignment and preferences · approach: Reward modeling for diverse preferences

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes R*, an LLM-driven evolutionary framework that evolves modular reward function structures with module-level crossover and tunes reward parameters via critic-based step-wise voting labels and preference learning.

Problem

Code-based reward function design with LLMs suffers from a huge design space, greedy exploitation of candidates, and unreliable LLM-set reward parameters, leading to low optimization efficiency for robotic control tasks.

Approach

Decouples reward design into structure evolution (LLM as mutation operator over a modular reward population with module-level crossover and an fitness archive) and parameter alignment (LLM-generated critic population votes on step-wise trajectory comparisons, and reward parameters are fit with a Bradley-Terry preference loss).

Key results

• R* significantly outperforms Eureka on 8 robotic manipulation tasks and matches or surpasses human-designed (Oracle) rewards in most tasks.
• Critic-based voting labeling achieves 100% accuracy on 100 manually labeled pairs, versus lower accuracy for LLM numerical comparison and GPT-4o VLM labeling.
• Over 50% of best-performing policies in most tasks come from module-level crossover individuals, which are only 4 of 16 population members.

Measured effects reported in the paper

Surprising findings

• Increasing crossover-generated individuals from 4 to 8 degraded performance, sometimes below the Eureka baseline.
• VLM-based labeling frequently outputs 0 (no preference), making it largely unusable for comparisons.

Limitations

• Only applicable when low-level state information is accessible; image-based inputs are not supported.
• LLM token overhead of crossover and reflection not optimized.
• Experiments limited to simulated manipulation tasks in Isaac Gym and Dexterity.

Experimental setup

Datasets: Isaac Gym tasks Bidextrous Manipulation (Dexterity) benchmark

Benchmarks: FrankaCabinet Shadow-Hand-Over Shadow-Hand-Swing-Cup Shadow-Hand-Scissors Allegro-Hand Shadow-Hand-Door-Open-Outward Shadow-Hand-Kettle Shadow-Hand-Pen

Metrics: success rate labeling accuracy

Baselines: Eureka Oracle human-designed reward Sparse reward VLM labeling (GPT-4o) LLM numerical labeling (GPT-4o)

Builds on: Eureka Text2Reward PEBBLE PPO Bradley-Terry preference model

Closest papers by meaning

• Efficient Language-instructed Skill Acquisition via Reward-Policy Co-Evolution (AAAI 2025 · similarity 0.91)
• Progress Reward Model for Reinforcement Learning via Large Language Models (NeurIPS 2025 · similarity 0.91)
• Text2Reward: Reward Shaping with Language Models for Reinforcement Learning (ICLR 2024 · similarity 0.90)
• RF-Agent: Automated Reward Function Design via Language Agent Tree Search (NeurIPS 2025 · similarity 0.89)
• Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilization (ICML 2024 · similarity 0.89)
• ORSO: Accelerating Reward Design via Online Reward Selection and Policy Optimization (ICLR 2025 · similarity 0.89)