Research map / Reinforcement learning / RL from feedback
Text2Reward: Reward Shaping with Language Models for Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Text2Reward uses GPT-4 to generate and iteratively refine shaped dense reward code from natural-language goals and a Pythonic environment abstraction, training PPO/SAC policies for manipulation and locomotion tasks.
Problem
Reward shaping for RL is costly and requires expert knowledge or data; learned reward models (IRL, preference learning) are opaque and data-hungry, and prior LLM reward-code work produces unshaped rewards with restricted APIs.
Approach
Generate free-form, shaped dense reward code with an LLM from a language goal, a compact Pythonic environment description, background function docs, and retrieved few-shot reward examples; execute and self-correct errors, then refine the code with natural-language human feedback on rollout videos.
Key results
- On 13 of 17 manipulation tasks, generated reward policies match or beat expert oracle rewards in success rate and convergence speed.
- Six novel locomotion behaviors (Hopper move forward/front flip/back flip; Ant move forward/lie down/wave leg) reach 94-100% human-judged success.
- Interactive feedback raises Stack Cube success from zero to near 100% within about two feedback iterations for few-shot code.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Text2Reward-generated dense reward codes instead of expert-written reward codes | RL policies on ManiSkill2 and MetaWorld manipulation tasks · 17 manipulation tasks | success rate and convergence speed | Mixed |
| use few-shot examples from the skills library instead of zero-shot generation (ablation) | Text2Reward · ManiSkill2 and MetaWorld manipulation tasks | downstream performance | Helped |
| use few-shot examples from the skills library instead of zero-shot generation (ablation) | Text2Reward · Turn Faucet and Open Doors | learning performance | Hurt |
| add human-feedback iterations to few-shot reward generation (ablation) | Text2Reward on Stack Cube · ManiSkill2 Stack Cube | success rate: from zero to one | Helped |
| add code-execution feedback to generated reward code (ablation) | Text2Reward · reward-code generation | error rate: from 10% to near zero | Helped |
Surprising findings
- Few-shot generated rewards outperform expert oracle rewards on some tasks such as Open Door, partly by omitting a final stabilization stage.
- Zero-shot rewards sometimes beat few-shot ones, because a poorly matched retrieved example can omit key reward terms.
Limitations
- Relies on proprietary GPT-4; open-source LLM results are only in an appendix and are weaker.
- Some tasks remain unsolved, and single-turn zero-shot code often cannot be improved much by feedback.
- Depends on an off-the-shelf perception pipeline (SAM plus depth) for real-robot object state.
Experimental setup
Datasets: MetaWorld ManiSkill2 Gym MuJoCo
Benchmarks: ManiSkill2 MetaWorld Gym MuJoCo
Metrics: task success rate convergence speed human-judged success rate on locomotion rollouts
Baselines: L2R Oracle expert-written reward
Builds on: Eureka Language to Rewards (L2R) PPO SAC Segment Anything Model
Closest papers by meaning
- Efficient Language-instructed Skill Acquisition via Reward-Policy Co-Evolution (AAAI 2025 · similarity 0.91)
- Progress Reward Model for Reinforcement Learning via Large Language Models (NeurIPS 2025 · similarity 0.91)
- Eureka: Human-Level Reward Design via Coding Large Language Models (ICLR 2024 · similarity 0.91)
- REvolve: Reward Evolution with Large Language Models using Human Feedback (ICLR 2025 · similarity 0.91)
- Dense Reward for Free in Reinforcement Learning from Human Feedback (ICML 2024 · similarity 0.90)
- R*: Efficient Reward Design via Reward Structure Evolution and Parameter Alignment Optimization with Large Language Models (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.