Research map / Reinforcement learning / Visual and representation RL
ORSO: Accelerating Reward Design via Online Reward Selection and Policy Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ORSO, which treats shaping-reward selection as online model selection (D3RB bandit-style) over LLM-generated reward candidates, trained with PPO on Isaac Gym tasks, to find good shaped rewards within a fixed interaction budget.
Problem
Choosing effective shaping reward functions from a candidate set is computationally expensive because each candidate requires lengthy RL training to evaluate under the task reward.
Approach
Frame shaping-reward selection as an online model selection problem where each candidate reward (with its policy) is a base learner, and allocate training budget with a regret-balancing algorithm (D3RB) so that compute concentrates on promising rewards, with regret guarantees.
Key results
- ORSO achieves 56% higher normalized task reward than naive selection at the highest interaction budget.
- ORSO reaches human-level performance with up to 8x fewer GPUs/less compute than naive selection.
- ORSO's normalized regret can become negative, meaning it finds rewards outperforming human-designed ones on average.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ORSO instead of naive reward selection at the highest interaction budget | Reward design with PPO · Continuous control tasks, averaged across tasks | normalized task reward: 56% higher | Helped |
| Use ORSO rather than naive selection | Reward design with PPO · Human-level performance across continuous control tasks | number of GPUs: 8 × fewer GPUs | Helped |
| Use D³RB rather than other selection algorithms (ablation) | ORSO · Continuous control tasks | average normalized task reward | Helped |
| Use Exp3 rather than greedy selection algorithms UCB, ETC, and EG (ablation) | ORSO · Continuous control tasks | average normalized task reward | Helped |
| Use EG rather than naive reward selection (ablation) | ORSO · Continuous control tasks | average normalized task reward | Helped |
| Use ETC rather than naive reward selection (ablation) | ORSO · Continuous control tasks | average normalized task reward | Helped |
| Use D³RB with a fixed reward set of 48 or 96 candidates, without iterative improvement (ablation) | ORSO · Ant | task reward | Helped |
| Use ε-greedy, ETC, or UCB with a fixed reward set of 48 or 96 candidates, without iterative improvement (ablation) | ORSO · Ant | task reward | Hurt |
| Use Exp3 or uniform exploration with a fixed reward set of 48 or 96 candidates, without iterative improvement (ablation) | ORSO · Ant | task reward | Hurt |
| Use 8 reward functions rather than other explored set sizes (ablation) | ORSO · Continuous control tasks | task reward | Helped |
Surprising findings
- Simple selection strategies such as epsilon-greedy and explore-then-commit already substantially outperform naive uniform selection.
- On average ORSO matches or exceeds expert-designed rewards on several tasks.
Limitations
- Relies on a predefined hand-crafted task reward, which is hard to specify for qualitative tasks.
- Experiments explored at most 16 reward functions with resampling and up to 96 without; larger sets left for future work.
- Cannot be parallelized across GPUs in the current formulation.
Experimental setup
Datasets: Isaac Gym simulated tasks (Cartpole, BallBalance, Ant, Humanoid, AllegroHand, ShadowHand)
Benchmarks: Isaac Gym Cartpole Isaac Gym BallBalance Isaac Gym Ant Isaac Gym Humanoid Isaac Gym AllegroHand Isaac Gym ShadowHand
Metrics: normalized task reward time to human-level performance normalized regret
Baselines: No Design (task reward) Human-engineered rewards Naive selection Eureka explore-then-commit epsilon-greedy UCB Exp3
Builds on: Eureka D3RB PPO Exp3 CleanRL
Closest papers by meaning
- Automatic Reward Shaping from Confounded Offline Data (ICML 2025 · similarity 0.90)
- Action-Dependent Optimality-Preserving Reward Shaping (ICML 2025 · similarity 0.89)
- R*: Efficient Reward Design via Reward Structure Evolution and Parameter Alignment Optimization with Large Language Models (ICML 2025 · similarity 0.89)
- Highly Efficient Self-Adaptive Reward Shaping for Reinforcement Learning (ICLR 2025 · similarity 0.88)
- STARC: A General Framework For Quantifying Differences Between Reward Functions (ICLR 2024 · similarity 0.88)
- Reward Shaping for Reinforcement Learning with An Assistant Reward Agent (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.