Research map / Reinforcement learning / MDPs and planning
PIGDreamer: Privileged Information Guided World Models for Safe Partially Observable Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PIGDreamer, a safe model-based RL method that uses privileged information during training (asymmetric actor-critic, privileged representation alignment and predictors) on top of DreamerV3 with a Lagrangian constraint, grounded in a new ACPOMDP framework.
Problem
Safe reinforcement learning under partial observability, where the agent must identify risks and rewards from partial observations; existing world-model approaches underuse privileged information available during training.
Approach
Introduces ACPOMDPs, where critics access underlying states, and PIGDreamer trains a naive and privileged world model together, aligning the observation-based representation to an oracle representation built from privileged info, and giving predictors and critics privileged inputs while the actor only sees observations.
Key results
- On the Guard benchmark, PIGDreamer reaches average reward 10.68 vs 3.38 for Scaffolder(Lag) and 4.52 for SafeDreamer, with average cost 4.59.
- Compared to the model-free CPO, PPO_Lag, TRPO_Lag and FOCOPS on Safety-Gymnasium tasks, PIGDreamer averages reward 14.5 with cost 6.0 under a cost limit of 2 (baselines at limit 25).
- PIGDreamer adds about 22.4% average training time over SafeDreamer, versus 69% for Scaffolder.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PIGDreamer instead of SafeDreamer | DreamerV3 on the Guard benchmark · Guard benchmark (Average) | Reward: 4.52 -> 10.68 | Helped |
| Use PIGDreamer instead of SafeDreamer | DreamerV3 on the Guard benchmark · Guard benchmark (Average) | Cost: 4.24 -> 4.59 | Mixed |
| Use PIGDreamer instead of the model-free baselines | Safety-Gymnasium · Safety-Gymnasium (Average) | Reward: 11.3 -> 14.5 | Helped |
| Use PIGDreamer instead of the model-free baselines | Safety-Gymnasium · Safety-Gymnasium (Average) | Cost: 28.3 -> 6.0 | Helped |
| Replace Distill with PIGDreamer | SafeDreamer-based method · SafetyPointGoal2 | Reward: 6.59 -> 11.61 | Helped |
| Replace Distill with PIGDreamer | SafeDreamer-based method · SafetyPointGoal2 | Cost: 4.54 -> 1.31 | Helped |
| Replace Distill with PIGDreamer | SafeDreamer-based method · SafetyRacecarGoal1 | Reward: 10.21 -> 10.99 | Helped |
| Replace Distill with PIGDreamer | SafeDreamer-based method · SafetyPointPush1 | Cost: 12.37 -> 1.15 | Helped |
| Replace Twisted Imagination with Nested Latent Imagination (ablation) | PIGDreamer on SafetyPointGoal2 · SafetyPointGoal2 | Reward: 13.59 -> 10.79 | Hurt |
| Replace Twisted Imagination with Nested Latent Imagination (ablation) | PIGDreamer on SafetyCarGoal1 · SafetyCarGoal1 | Cost: 0.43 -> 0.64 | Mixed |
| Replace Twisted Imagination with Nested Latent Imagination (ablation) | PIGDreamer on SafetyRacecarGoal1 · SafetyRacecarGoal1 | Reward: 11.38 -> 13.99 | Helped |
| Remove privileged representation alignment (ablation) | PIGDreamer · Safety-Gymnasium | Reward | Hurt |
Surprising findings
- Privileged variants such as Distill-Dreamer(Lag) and Informed-Dreamer(Lag) can underperform unprivileged SafeDreamer, which the authors attribute to an information gap the agent cannot compensate for.
- On PointButton1, PIGDreamer does not outperform SafeDreamer even with privileged information, which the authors attribute to the privileged information yielding no information gain.
Limitations
- Privileged information does not always help; no gain on PointButton1.
- Results based on three runs across five tasks, with aggregate metrics reported with wide bootstrap intervals.
- Theoretical result (ACPOMDP value dominance) relies on a loosely specified argument with informal steps in the proof.
Experimental setup
Datasets: Safety-Gymnasium Guard
Benchmarks: Safety-Gymnasium Guard
Metrics: episodic reward episodic cost IQM training time
Baselines: SafeDreamer LAMBDA Safe-SLAC Scaffolder(Lag) Informed-Dreamer(Lag) Distill-Dreamer(Lag) CPO PPO_Lag TRPO_Lag FOCOPS
Builds on: DreamerV3 SafeDreamer Scaffolder Informed-Dreamer Augmented Lagrangian
Closest papers by meaning
- Provable Partially Observable Reinforcement Learning with Privileged Information (NeurIPS 2024 · similarity 0.92)
- Provable Representation with Efficient Planning for Partially Observable Reinforcement Learning (ICML 2024 · similarity 0.88)
- Provably Efficient Partially Observable Risk-sensitive Reinforcement Learning with Hindsight Observation (ICML 2024 · similarity 0.88)
- Mitigating Partial Observability in Sequential Decision Processes via the Lambda Discrepancy (NeurIPS 2024 · similarity 0.87)
- SafeDreamer: Safe Reinforcement Learning with World Models (ICLR 2024 · similarity 0.87)
- Safeguarded Progress in Reinforcement Learning: Safe Bayesian Exploration for Control Policy Synthesis (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.