Research map / Reinforcement learning / RL from feedback
Progress Reward Model for Reinforcement Learning via Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PRM4RL, which uses GPT-4o to decompose robot tasks into subtasks and generate a progress function, then shapes rewards with a potential-based Progress Reward Model for SAC/PPO on MetaWorld and ManiSkill.
Problem
RL with sparse rewards on long-horizon robotic manipulation tasks; existing LLM planners lack low-level dense reward optimization and LLM rewarders need heavy human feedback or evolutionary search.
Approach
Combine LLM subtask decomposition (with a subtask determination function and a progress function written as Python code) with potential-based reward shaping, using the progress function as the potential, plus subtask-prior embeddings appended to the state.
Key results
- On MW-pick-out-of-hole and MS-LiftPegUpright, baselines achieve under 15% success while PRM4RL reaches nearly 100%.
- Ablation: using progress function directly as reward (w.o. potential reward) causes success rate to rise then decline despite high episode reward.
- Generalization to unseen MetaWorld tasks: PRM4RL shows the least performance loss among compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine subtask planning with the Progress Reward Model (PRM) | PRM4RL · MetaWorld and ManiSkill | success rate | Helped |
| Use PRM4RL instead of comparison methods on complex long-term tasks | PRM4RL · MW-pick-out-of-hole; MS-LiftPegUpright | success rate: less than 15% -> nearly 100% | Helped |
| Add PRM reward shaping | PRM4RL · MetaWorld | convergence speed | Helped |
| Decompose complex tasks into subtasks before generating rewards | PRM4RL · MetaWorld and ManiSkill | task performance | Helped |
| Replace PRM reward with oracle reward while retaining the subtask prior (ablation) | PRM4RL · MetaWorld | success rate | Hurt |
| Add the subtask prior while retaining PRM reward (ablation) | PRM4RL · MetaWorld | convergence speed | Helped |
| Remove the subtask prior while retaining PRM reward (ablation) | PRM4RL · unseen MetaWorld tasks | performance | Hurt |
| Use the progress function directly as the reward instead of potential-based shaping (ablation) | PRM4RL · MetaWorld | success rate | Hurt |
| Add the subtask prior when using oracle reward (ablation) | PRM4RL without PRM reward · MetaWorld | performance | Mixed |
Surprising findings
- Directly using the progress function as reward yields high episode reward while success rate paradoxically declines during training.
- Replacing the PRM reward with the oracle reward while keeping the subtask prior gives little or no gain, sometimes a slight decrease.
Limitations
- LLM hallucination can affect generated planning and progress code.
- Applicability outside robotics is unexplored.
- Relies on environment state descriptions to write determination and progress functions.
Experimental setup
Datasets: MetaWorld ManiSkill
Benchmarks: MetaWorld ManiSkill
Metrics: success rate convergence speed LLM calls
Baselines: ELLM Text2Reward Oracle reward
Builds on: potential-based reward shaping PAL Text2Reward ELLM SimCSE
Closest papers by meaning
- Text2Reward: Reward Shaping with Language Models for Reinforcement Learning (ICLR 2024 · similarity 0.91)
- On the Modeling Capabilities of Large Language Models for Sequential Decision Making (ICLR 2025 · similarity 0.91)
- R*: Efficient Reward Design via Reward Structure Evolution and Parameter Alignment Optimization with Large Language Models (ICML 2025 · similarity 0.91)
- Dense Reward for Free in Reinforcement Learning from Human Feedback (ICML 2024 · similarity 0.90)
- Efficient Language-instructed Skill Acquisition via Reward-Policy Co-Evolution (AAAI 2025 · similarity 0.90)
- Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Tasks (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.