Research map / Reinforcement learning / MDPs and planning
Provable Partially Observable Reinforcement Learning with Privileged Information
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretically analyzes two practical privileged-information paradigms for partially observable RL (expert distillation and asymmetric actor-critic, plus CTDE for POSGs), giving polynomial/quasi-polynomial sample and computational guarantees under a deterministic filter condition and observability.
Problem
Partially observable RL with privileged state information (e.g., simulator states, CTDE in MARL) is widely used empirically, but its theoretical benefits and the efficiency of practical paradigms (expert distillation, asymmetric actor-critic) are poorly understood, and prior provable methods rely on
Approach
Shows expert distillation fails in general POMDPs but succeeds under a weaker-than-known "deterministic filter" condition with polynomial sample and computation; develops a belief-weighted optimistic asymmetric actor-critic using a provable approximate belief learner (via model truncation) giving polynomial sample and quasi-polynomial time, and extends the ideas to POSGs with information sharing.
Key results
- On deterministic POMDP and block MDP cases, expert policy distillation achieves the highest reward (e.g., 7.26 ± 0.68 vs 7.10 ± 0.48 for asymmetric optimistic NPG in Case 2 of deterministic POMDPs).
- Expert distillation under the deterministic filter condition achieves polynomial sample and computational complexity, including for unknown decoding length.
- Asymmetric optimistic actor-critic with approximate belief learning achieves polynomial sample and quasi-polynomial time complexity for observable POMDPs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use expert policy distillation instead of Vanilla AAC | POMDP policies · Deterministic POMDP Case 1 | reward: 3.25 \pm 0.65 -> 3.33 \pm 0.62 | No clear effect |
| Use expert policy distillation instead of Vanilla AAC | POMDP policies · Deterministic POMDP Case 2 | reward: 6.41 \pm 0.91 -> 7.26 \pm 0.68 | Helped |
| Use expert policy distillation instead of Vanilla AAC | POMDP policies · Deterministic POMDP Case 3 | reward: 3.10 \pm 0.38 -> 3.25 \pm 0.33 | No clear effect |
| Use expert policy distillation instead of Vanilla AAC | POMDP policies · Deterministic POMDP Case 4 | reward: 5.87 \pm 0.58 -> 6.54 \pm 0.58 | Helped |
| Use expert policy distillation instead of Vanilla AAC | POMDP policies · Block MDP Case 1 | reward: 3.08 \pm 0.43 -> 3.37 \pm 0.41 | Helped |
| Use expert policy distillation instead of Vanilla AAC | POMDP policies · Block MDP Case 2 | reward: 5.70 \pm 0.46 -> 6.67 \pm 0.54 | Helped |
| Use expert policy distillation instead of Vanilla AAC | POMDP policies · Block MDP Case 3 | reward: 2.97 \pm 0.32 -> 3.37 \pm 0.22 | Helped |
| Use expert policy distillation instead of Vanilla AAC | POMDP policies · Block MDP Case 4 | reward: 5.33 \pm 0.19 -> 6.44 \pm 0.36 | Helped |
Surprising findings
- Expert policy distillation can be strictly suboptimal even with infinite data in POMDPs satisfying observability (gamma-observable with H=1)
- Vanilla asymmetric actor-critic with state-only critics is biased, requiring history-and-state-conditioned critics
Limitations
- Assumes exact state information is available during training
- Theory is primarily tabular; function approximation only partially covered
Experimental setup
Benchmarks: randomly generated deterministic POMDPs randomly generated block MDPs randomly generated general POMDPs
Metrics: episode reward
Baselines: vanilla asymmetric actor-critic asymmetric Q-learning asymmetric optimistic value iteration
Builds on: asymmetric actor-critic observable POMDP quasi-polynomial planning hindsight observable MDPs k-step decodable POMDPs natural policy gradient
Closest papers by meaning
- PIGDreamer: Privileged Information Guided World Models for Safe Partially Observable Reinforcement Learning (ICML 2025 · similarity 0.92)
- Provable Representation with Efficient Planning for Partially Observable Reinforcement Learning (ICML 2024 · similarity 0.90)
- Mitigating Partial Observability in Sequential Decision Processes via the Lambda Discrepancy (NeurIPS 2024 · similarity 0.90)
- A Theoretical Justification for Asymmetric Actor-Critic Algorithms (ICML 2025 · similarity 0.89)
- Provably Efficient Partially Observable Risk-sensitive Reinforcement Learning with Hindsight Observation (ICML 2024 · similarity 0.88)
- Dual Critic Reinforcement Learning under Partial Observability (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.