Research map / Large language models / Alignment and preferences
Eureka: Human-Level Reward Design via Coding Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Eureka uses GPT-4 to zero-shot write reward code from raw environment source, then improves it via evolutionary search with reward reflection, matching or beating expert rewards on 83% of 29 IsaacGym tasks and enabling pen spinning with curriculum learning.
Problem
Designing reward functions for RL is labor-intensive and often suboptimal; the paper asks whether coding LLMs can automatically generate human-level reward code for complex low-level dexterous manipulation tasks.
Approach
Use raw environment source code as LLM context to zero-shot generate executable reward code, then run evolutionary search in which the LLM mutates the best reward using a textual reward reflection built from per-component reward statistics collected during RL training.
Key results
- Eureka outperforms expert human rewards on 83% of 29 tasks with an average normalized improvement of 52%, exceeding or matching human on all Isaac tasks and 15 of 20 Dexterity tasks.
- Eureka without reward reflection reduces average normalized score on Isaac tasks by 28.6%.
- With curriculum learning, Eureka fine-tuning enables a simulated Shadow Hand to spin a pen for multiple cycles, while scratch and pre-trained policies complete none.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Eureka-generated rewards instead of the human reward | Mujoco Humanoid · Mujoco Humanoid | task score: 5.92 -> 7.68 | Helped |
| Add iterative evolutionary reward search rather than spending the same sample budget on initial samples (ablation) | Eureka · Isaac and Dexterity | task performance | Helped |
| Add reward reflection to feedback instead of providing only task-metric snapshots (ablation) | Eureka · Isaac tasks | average normalized score: 28.6% | Helped |
| Use GPT-4 rather than GPT-3.5 as the reward-design LLM (ablation) | Eureka · Isaac tasks | task performance | Helped |
| Add curriculum pre-training before fine-tuning on the pen-spinning sequence (ablation) | Eureka pen-spinning policy · Pen spinning | completed spinning cycles | Helped |
| Initialize Eureka with the human-written reward (ablation) | Eureka · Selected Dexterity tasks | task performance | Helped |
| Replace automatic reward reflection with human textual feedback | Eureka-HF · Humanoid running | human preference | Helped |
Surprising findings
- Eureka rewards are sometimes negatively correlated with human rewards yet perform significantly better, suggesting counter-intuitive reward design principles.
- Human-initialized Eureka improves even when human rewards are highly sub-optimal, suggesting humans know relevant state variables but are less skilled at composing rewards.
Limitations
- Relies on GPT-4 API and a fixed, human-chosen task fitness function for evaluation and feedback.
- Eureka rewards show less smooth RL training curves than human rewards, as PPO hyperparameters were tuned for human rewards.
- Baseline L2R required substantial manual engineering of templates and reward APIs, making comparison imperfect.
Experimental setup
Datasets: IsaacGym environments (9 Isaac tasks) Dexterity benchmark (20 bimanual tasks) OpenAI Gym Humanoid (MuJoCo)
Benchmarks: IsaacGym Isaac tasks Dexterity MuJoCo Humanoid pen spinning
Metrics: human normalized score success rate task fitness probability of improvement user preference
Baselines: Human Sparse L2R Eureka w.o. Evolution (32 Samples) Eureka (No Reward Reflection) GPT-3.5 Eureka
Builds on: Language-to-Rewards PPO IsaacGym GPT-4
Closest papers by meaning
- Efficient Language-instructed Skill Acquisition via Reward-Policy Co-Evolution (AAAI 2025 · similarity 0.91)
- Text2Reward: Reward Shaping with Language Models for Reinforcement Learning (ICLR 2024 · similarity 0.91)
- Learning Reward for Robot Skills Using Large Language Models via Self-Alignment (ICML 2024 · similarity 0.90)
- REvolve: Reward Evolution with Large Language Models using Human Feedback (ICLR 2025 · similarity 0.90)
- GROVE: A Generalized Reward for Learning Open-Vocabulary Physical Skill (CVPR 2025 · similarity 0.89)
- Progress Reward Model for Reinforcement Learning via Large Language Models (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.