Research map / Reinforcement learning / Visual and representation RL
Reward Machines for Deep RL in Noisy and Uncertain Environments
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Frames RL with Reward Machines under noisy/uncertain proposition evaluation as a POMDP and proposes Naive, IBU, and TDM inference modules that use abstraction models, showing TDM matches an oracle while naive aggregation fails.
Problem
Reward Machine-based RL assumes exact labelling functions for propositions, but real-world perception is partial and noisy, so the reward-relevant vocabulary is uncertain.
Approach
Model the noisy RM environment as a POMDP over state space S x U, and decouple RM-state inference (from noisy abstraction model outputs) from decision-making; a temporal dependency modelling (TDM) abstraction model predicts a distribution over RM states directly, avoiding the correlated-error amplification that Naive and IBU suffer from.
Key results
- TDM achieves sample efficiency and final performance similar to the Oracle agent across domains without access to the ground-truth labelling function, while Recurrent PPO fails.
- TDM yields higher log-likelihood of the true RM state than Naive and IBU in all domains.
- GPT-4o as a zero-shot abstraction model is nearly as effective as a custom supervised model for Naive and IBU in Traffic Light.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use TDM inference instead of a recurrent memory-only policy | Reward Machine deep RL · Traffic Light, Kitchen, Colour Matching | total return | Helped |
| use TDM with a noisy abstraction model instead of an Oracle with the ground-truth labelling function | Reward Machine deep RL · Traffic Light, Kitchen, Colour Matching | sample efficiency and final performance | No clear effect |
| use TDM inference instead of Naive or IBU inference | RM state belief inference · Gold Mining, Traffic Light, Kitchen, Colour Matching | log-likelihood | Helped |
| use Naive inference with a noisy abstraction model | Reward Machine RL · Traffic Light | task safety | Hurt |
| use IBU inference with a noisy abstraction model | Reward Machine RL · Kitchen | task completion | Hurt |
| use IBU inference with a noisy abstraction model | Reward Machine RL · Traffic Light | task safety | Helped |
| use Naive inference with a noisy abstraction model | Reward Machine RL · Kitchen | task completion | Helped |
| use GPT-4o as a zero-shot abstraction model instead of a randomly initialized neural network | Naive and IBU inference · Traffic Light | RM state prediction accuracy | Helped |
| use GPT-4o as a zero-shot abstraction model instead of a supervised model trained from ground-truth data | Naive and IBU inference · Traffic Light | RM state prediction accuracy | No clear effect |
| account for temporal dependencies in abstraction-model predictions | RM state belief inference · Gold Mining, Kitchen | RM state belief accuracy | Helped |
Surprising findings
- IBU's independence assumption causes its belief to converge to near-certainty that chores are done in Kitchen, so the agent never enters the kitchen.
- Naive can run red lights in Traffic Light by driving in reverse because it cannot represent uncertainty.
Limitations
- TDM relies on a task-specific abstraction model that must be trained on enough data.
- Naive and IBU can produce dangerous or unintended outcomes under noisy abstraction models.
- Training uses ground-truth rewards, which may not be available in practice.
Experimental setup
Datasets: randomly generated episodes from Traffic Light, Kitchen, and Colour Matching environments
Benchmarks: Gold Mining Traffic Light (MiniGrid) Kitchen (MiniGrid) Colour Matching (MuJoCo)
Metrics: episodic return / sample efficiency log-likelihood of true RM state precision and recall of proposition classifiers
Baselines: Naive IBU Memory-only (Recurrent PPO) Oracle
Builds on: Reward Machines PPO Recurrent PPO Q-learning
Closest papers by meaning
- RM-R1: Reward Modeling as Reasoning (ICLR 2026 · similarity 0.88)
- R*: Efficient Reward Design via Reward Structure Evolution and Parameter Alignment Optimization with Large Language Models (ICML 2025 · similarity 0.88)
- Text2Reward: Reward Shaping with Language Models for Reinforcement Learning (ICLR 2024 · similarity 0.88)
- Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward (ICLR 2026 · similarity 0.88)
- REvolve: Reward Evolution with Large Language Models using Human Feedback (ICLR 2025 · similarity 0.88)
- How Far Can Unsupervised RLVR Scale LLM Training? (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.