Research map / Large language models / Alignment and preferences
Learning Reward for Robot Skills Using Large Language Models via Self-Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes learning robot reward functions by having an LLM (GPT-4) propose reward features and parameterization, then iteratively updating parameters via ranking-based self-alignment with the LLM's trajectory rankings, plus LLM-guided active parameter adjustment.
Problem
Learning reward functions for robot skills without human demonstrations or feedback; LLM-proposed reward functions are numerically imprecise and need grounding with environment feedback, and existing LLM reward-design methods rely on human feedback or costly evolutionary mutation.
Approach
Use the LLM as a ranking oracle over executed trajectories (described by reward features) and update reward parameters via Bayesian/Metropolis-Hastings preference learning to minimize ranking inconsistency between the LLM and the learned reward (self-alignment), with LLM reflection used only to adjust parameters when no discrepancy yet yields success.
Key results
- Self-alignment with active adjustment (SA+AA) reaches 0.73±0.16 success on Pick-Cube versus 0.01±0.02 for the fixed LLM reward, and 0.99 on Cabinet and 0.84 on Chair.
- On Isaac Gym Shadow Hand Open Door, max success rate reaches 100% versus 0.00% for Eureka, and 94.54% on Scissor versus 0.24% for Eureka.
- Token consumption on Cabinet is 15,633 for the proposed method versus 64,698 for Eureka; on Door it is 17,543 versus 1,284,924 (Eureka hit the 50-iteration limit).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply self-alignment reward updates | LLM-proposed reward · Pick-Cube and Peg-Insertion | success rate | Helped |
| use self-alignment updates on Text2Reward zero-shot rewards | Text2Reward reward functions · Pick-Cube and Push-Chair | success rate | Helped |
| use self-alignment reward learning instead of Eureka mutation | robot reward learning · Shadow Hand Open Door Outwards | max success rate: 0.00% -> 100.00% | Helped |
| use self-alignment reward learning instead of Eureka mutation | robot reward learning · Shadow Hand Open Scissor | max success rate: 0.24% -> 94.54% | Helped |
| use self-alignment reward learning instead of Eureka mutation | robot reward learning · Franka Cabinet | GPT tokens: 64,698 -> 15,633 | Helped |
| use self-alignment reward learning instead of Eureka mutation | robot reward learning · Shadow Hand Open Door Outwards | GPT tokens: 1,284,924 -> 17,543 | Helped |
| use self-alignment reward learning instead of Eureka mutation | robot reward learning · Shadow Hand Open Scissor | GPT tokens: 1,338,778 -> 130,969 | Helped |
| add self-alignment to fixed LLM reward (ablation) | Pick-Cube · Pick-Cube (16M steps) | success rate: 0.01±0.02 -> 0.30±0.15 | Helped |
| add active parameter adjustment to fixed LLM reward (ablation) | Pick-Cube · Pick-Cube (16M steps) | success rate: 0.01±0.02 -> 0.47±0.49 | Helped |
| combine self-alignment and active adjustment (ablation) | LLM-proposed reward · Pick-Cube (16M steps) | success rate: 0.01±0.02 -> 0.73±0.16 | Helped |
| add self-alignment to fixed LLM reward (ablation) | Open-Cabinet-Door · Open-Cabinet-Door (8M steps) | success rate: 0.97±0.05 -> 0.97±0.03 | No clear effect |
| add active parameter adjustment to fixed LLM reward (ablation) | Open-Cabinet-Door · Open-Cabinet-Door (8M steps) | success rate: 0.97±0.05 -> 0.99±0.02 | No clear effect |
Surprising findings
- Fixed LLM-proposed rewards achieved nearly zero success on Pick-Cube and Peg-Insertion, showing numerical parameter imprecision is a major failure mode.
- Eureka's max success rate was 0% on Door and 0.24% on Scissor under the authors' reproduction with the newer GPT-4 model.
Limitations
- Uses only the final-step reward features for feedback, so tasks where trajectory shape matters (e.g., S-shaped walking) are not handled.
- Assumes all rankings are equally better or worse, which is suboptimal for sparse-reward tasks like peg insertion.
- LLM ranking can be wrong with subtle numerical differences and is sensitive to ambiguous task descriptions.
Experimental setup
Datasets: ManiSkill2 simulated trajectories Isaac Gym simulated trajectories
Benchmarks: ManiSkill2 Isaac Gym
Metrics: success rate tokens consumed max success rate
Baselines: Oracle reward Fixed LLM reward (CoT) Text2Reward Eureka
Builds on: Eureka Text2Reward Inverse reinforcement learning Bradley-Terry preference model APReL
Closest papers by meaning
- Efficient Language-instructed Skill Acquisition via Reward-Policy Co-Evolution (AAAI 2025 · similarity 0.91)
- Eureka: Human-Level Reward Design via Coding Large Language Models (ICLR 2024 · similarity 0.90)
- Text2Reward: Reward Shaping with Language Models for Reinforcement Learning (ICLR 2024 · similarity 0.89)
- Self-Rewarding Language Models (ICML 2024 · similarity 0.89)
- Progress Reward Model for Reinforcement Learning via Large Language Models (NeurIPS 2025 · similarity 0.89)
- SALMON: Self-Alignment with Instructable Reward Models (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.