Research map / Reinforcement learning / Visual and representation RL
Reward Translation via Reward Machine in Semi-Alignable MDPs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes neural reward translation (NRT), which transfers dense rewards across semi-alignable MDPs by matching reward machines (hand-designed or LLM-generated) with graph matching, and shows gains on NChain, Miniworld, and MuJoCo sparse-reward tasks.
Problem
Cross-domain reward transfer fails when source and target MDPs are incompatible (non-pairable, non-time-alignable), so existing alignment-based imitation methods do not apply.
Approach
Define semi-alignable MDPs via injections between abstract goal and skill spaces, and link them indirectly through reward machines; graph matching between source and target reward machines lets a potential-based shaped reward from the source task be transferred to the target task.
Key results
- On Miniworld-Sign with sparse reward, PPO and PPO-RM fail while PPO-NRT(Reward) and PPO-NRT(RM+Reward) substantially improve learning.
- On MuJoCo Ant-v3 with sparse reward, augmenting observations with reward machine state alone is insufficient, while transferred rewards from NChain markedly improve learning; PPO-NRT(Reward) performs best.
- LLM-generated reward machine for MuJoCo keeps the same structure as the hand-designed one (qualitative comparison).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| transfer rewards from the Text-Sign reward machine | PPO in Miniworld-Sign · Miniworld-Sign | performance | Helped |
| add reward-machine state to observations without transferring rewards (ablation) | PPO in Miniworld-Sign · Miniworld-Sign | performance | No clear effect |
| transfer rewards from the NChain reward machine | PPO in Ant-v3 · Ant-v3 | performance | Helped |
| add reward-machine state to observations without transferring rewards (ablation) | PPO in Ant-v3 · Ant-v3 | performance | No clear effect |
| use transferred rewards without adding reward-machine state to observations (ablation) | PPO in Ant-v3 · Ant-v3 | performance | Helped |
| add reward-machine state to the observation alongside transferred rewards (ablation) | PPO-NRT(Reward) in HalfCheetah-v3 · HalfCheetah-v3 | performance | No clear effect |
| use naive per-item reward shaping instead of transferred reward machines (ablation) | PPO in Miniworld-Sign · Miniworld-Sign | performance | Helped |
| use transferred reward machines instead of naive per-item reward shaping | PPO in Miniworld-Sign · Miniworld-Sign | performance | Helped |
| transfer rewards from the Text-Sign reward machine | PPO in Sign-with-Trap · Sign-with-Trap | performance | Helped |
| transfer rewards from the NChain reward machine | DQN in NChain(n=9) · NChain(n=9) | performance | Helped |
| use transferred rewards | PPO in Hopper-v3 · Hopper-v3 | performance | Helped |
| use transferred rewards | PPO in HalfCheetah-v3 · HalfCheetah-v3 | performance | Helped |
Surprising findings
- Augmenting observations with reward machine state alone does not help in extremely sparse reward settings; only transferred rewards help.
Limitations
- Constructing appropriate reward machines remains challenging, relying on human design or LLM generation.
- Deciphering relationships in complex tasks is still difficult, as acknowledged by the authors.
- Code is an early-stage version that may lack documentation.
Experimental setup
Datasets: NChain Text-Sign Miniworld-Sign MuJoCo HalfCheetah-v3 MuJoCo Hopper-v3 MuJoCo Ant-v3
Benchmarks: Miniworld-Sign MuJoCo Ant-v3 MuJoCo HalfCheetah-v3 MuJoCo Hopper-v3 NChain
Metrics: learning curve / episodic return
Baselines: PPO PPO-RM PPO-NRT(Reward) PPO-NRT(RM+Reward) SAC DQN
Builds on: reward machines QRM CRM PPO DAIL
Closest papers by meaning
- Active Reward Modeling: Adaptive Preference Labeling for Large Language Model Alignment (ICML 2025 · similarity 0.88)
- Learning Reward for Robot Skills Using Large Language Models via Self-Alignment (ICML 2024 · similarity 0.88)
- Text2Reward: Reward Shaping with Language Models for Reinforcement Learning (ICLR 2024 · similarity 0.87)
- Reward Machines for Deep RL in Noisy and Uncertain Environments (NeurIPS 2024 · similarity 0.87)
- Approximated Variational Bayesian Inverse Reinforcement Learning for Large Language Model Alignment (AAAI 2025 · similarity 0.87)
- ReDit: Reward Dithering for Improved LLM Policy Optimization (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.