Research map / Reinforcement learning / Policy gradient and actor-critic
STARC: A General Framework For Quantifying Differences Between Reward Functions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces STARC metrics, a family of reward-function pseudometrics built from canonicalisation (removing potential shaping and S'-redistribution), normalisation, and a distance, proven sound and complete w.r.t. worst-case regret and shown to outperform EPIC and DARD empirically.
Problem
Quantifying differences between reward functions in a principled way so that a small distance guarantees low worst-case regret (and vice versa), since L2 distance and test-set or rollout-based evaluations are unsatisfactory for evaluating reward learning.
Approach
Define a class of pseudometrics (STARC) that first canonicalises rewards to remove potential shaping and S'-redistribution (linearly, via Value-Adjusted Levelling), then normalises by a norm to ignore positive scaling, then measures distance with an admissible metric; these are proven to induce both upper and lower bounds on worst-case regret and to be bilipschitz unique among such metrics.
Key results
- VAL-canonicalised STARC metric (L2 normalisation and distance) reaches correlation 0.856 with regret vs 0.778 for EPIC and 0.782 for DARD in 32-state random MDPs.
- Using L1 for normalisation and distance raises the VAL-based STARC correlation from 0.856 to 0.873.
- In the Reacher environment, estimated STARC distances rank rewards as expected (PotentialShaped 0.897, SecondPeak 1.257, Random 1.378, Negative 1.706), though the GroundTruth-PotentialShaped true distance of 0 is estimated as about 0.9.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VAL instead of EPIC canonicalisation with L2 normalisation and distance (ablation) | Reward-function distance metrics · Random MDPs | correlation with regret: 0.778 -> 0.856 | Helped |
| Use L1 instead of L2 for both normalisation and distance with VAL canonicalisation (ablation) | STARC-VAL · Random MDPs | correlation with regret: 0.856 to 0.873 | Helped |
| Use weighted L-infinity normalisation and unweighted L-infinity distance instead of L2 for both (ablation) | EPIC-based metric · Random MDPs | correlation with regret: 0.778 to 0.052 | Hurt |
| Use VAL instead of EPIC canonicalisation (ablation) | Reward-function distance metrics · Random MDPs | correlation with regret: 40 out of 42 cases | Helped |
| Use VALPotential canonicalisation with a transition-weighted norm (ablation) | Reward-function distance metrics · Random MDPs | correlation with regret | No clear effect |
Surprising findings
- EPIC's canonicalisation with a weighted L_inf normalisation and unweighted L_inf distance drops its regret correlation from 0.778 to 0.052.
- VALPotential performs as well as VAL only when a tau-weighted norm is used.
Limitations
- Theoretical results assume finite state and action spaces; continuous environments only covered empirically.
- Experiments are indicative rather than conclusive about which STARC metric works best in practice.
- Uses a fairly strong definition of regret; weaker criteria might allow more permissive metrics.
Experimental setup
Datasets: randomly generated MDPs (32 states, 4 actions) Reacher (MuJoCo)
Benchmarks: randomly generated MDPs Reacher
Metrics: Pearson correlation with worst-case regret estimated distance
Baselines: EPIC DARD MinimalPotential VALPotential rollout regret
Builds on: EPIC DARD potential shaping
Closest papers by meaning
- The Perils of Optimizing Learned Reward Functions: Low Training Error Does Not Guarantee Low Regret (ICML 2025 · similarity 0.89)
- Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences (ICML 2024 · similarity 0.88)
- ORSO: Accelerating Reward Design via Online Reward Selection and Policy Optimization (ICLR 2025 · similarity 0.88)
- To the Max: Reinventing Reward in Reinforcement Learning (ICML 2024 · similarity 0.88)
- Correlated Proxies: A New Definition and Improved Mitigation for Reward Hacking (ICLR 2025 · similarity 0.88)
- Reward-Consistent Dynamics Models are Strongly Generalizable for Offline Reinforcement Learning (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.