Research map / Large language models / Alignment and preferences
REvolve: Reward Evolution with Large Language Models using Human Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes REvolve, an evolutionary framework where GPT-4 acts as genetic operator to evolve Python reward functions, using human pairwise preferences (Elo) as fitness, and tests on autonomous driving, humanoid, and Adroit hand tasks.
Problem
Reward design for RL in tasks where good behavior is tacit and hard to quantify (e.g., autonomous driving, dexterous manipulation), where greedy LLM-based reward search (Eureka) lacks a fitness signal beyond explicit metrics.
Approach
Treat LLM reward design as genetic programming over a population of reward programs with island model, mutation, crossover, migration and selection, where fitness comes from human pairwise video preferences converted to Elo scores plus natural-language feedback fed back to GPT-4.
Key results
- On autonomous driving, REvolve achieved human-preference Elo 1575, second only to human driving (1586) and above Eureka (1499) and Expert Designed (1397).
- On humanoid locomotion, REvolve reached Elo 1586 vs Eureka 1557 and Expert Designed 1407.
- On generalization Env 1, REvolve scored 0.86 fitness with 1146 episodic steps versus Eureka 0.73 and Expert Designed 0.71.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use population-based evolutionary reward search with human preferences and natural-language feedback instead of greedy search | REvolve vs. Eureka · Autonomous Driving, Humanoid Locomotion, and Adroit Hand Manipulation | Elo score: 1575 vs 1499; 1586 vs 1557; 1594 vs 1522 | Helped |
| Use human feedback instead of automatically generated feedback | REvolve · Autonomous Driving | Elo score: 1575 vs 1529 | Helped |
| Use human feedback instead of automatically generated feedback | REvolve · Humanoid Locomotion | Elo score: 1586 vs 1514 | Helped |
| Use human feedback instead of automatically generated feedback | REvolve · Adroit Hand Manipulation | Elo score: 1594 vs 1524 | Helped |
| Combine mutation and crossover rather than using crossover only or mutation only (ablation) | REvolve · Autonomous Driving | fitness | Helped |
| Add mutation to crossover-only evolution (ablation) | REvolve · Autonomous Driving | fitness | Helped |
| Add crossover to mutation-only evolution (ablation) | REvolve · Autonomous Driving | fitness | Helped |
| Train from scratch with REvolve-designed rewards in unseen AirSim environments rather than using expert-designed rewards | REvolve reward functions · AirSim Env 1 and Env 2 (unseen) | fitness: 0.71 ± 0.03 vs 0.86 ± 0.03; 0.48 ± 0.02 vs 0.68 ± 0.04 | Helped |
Surprising findings
- REvolve Auto (automatic feedback) performed on par with Eureka with human feedback on humanoid and Adroit hand tasks, attributed to sparse automatic feedback in those tasks.
Limitations
- Relies on closed-source GPT-4, with reward design gap to open-source LLMs
- Training multiple deep RL policies is computationally expensive and slow
- Only two random seeds used due to compute cost
Experimental setup
Datasets: AirSim Neighborhood and Coastline environments MuJoCo Humanoid MuJoCo Adroit Hand
Benchmarks: AirSim autonomous driving Humanoid locomotion (MuJoCo) Adroit Hand door opening (MuJoCo) AirSim generalization Env 1 and Env 2
Metrics: task fitness score human Elo score episodic steps
Baselines: Eureka Eureka Auto REvolve Auto Expert Designed Text2Reward Human Driving
Builds on: Eureka Text2Reward genetic programming island model Soft Actor-Critic
Closest papers by meaning
- Text2Reward: Reward Shaping with Language Models for Reinforcement Learning (ICLR 2024 · similarity 0.91)
- Efficient Language-instructed Skill Acquisition via Reward-Policy Co-Evolution (AAAI 2025 · similarity 0.90)
- Dense Reward for Free in Reinforcement Learning from Human Feedback (ICML 2024 · similarity 0.90)
- Eureka: Human-Level Reward Design via Coding Large Language Models (ICLR 2024 · similarity 0.90)
- Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning (NeurIPS 2024 · similarity 0.89)
- SELF-EVOLVED REWARD LEARNING FOR LLMS (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.