Research map / Reinforcement learning / MDPs and planning
Mitigating Partial Observability in Sequential Decision Processes via the Lambda Discrepancy
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the lambda-discrepancy, the difference between two TD(lambda) value estimates with different lambda, as a detector and learning signal for partial observability, and uses it as an auxiliary loss for recurrent PPO on large POMDPs.
Problem
Reinforcement learning typically assumes a Markovian state representation; when state is only partially observable, the agent needs to learn a state representation and detect whether it has found a Markovian one, without access to the hidden state space.
Approach
Measure non-Markovianity as the norm of the difference between value functions from TD(0) and TD(1) (or other lambda pairs). This difference is provably zero for MDPs and almost always non-zero for POMDPs, so minimizing it (as an auxiliary loss on two recurrent value networks) pushes the GRU latent state toward a Markovian representation.
Key results
- The lambda-discrepancy auxiliary loss improves final performance and learning rate over recurrent and memoryless PPO in all four large POMDPs.
- In RockSample (15,15), baseline agents learn to exit immediately for +10 reward and never improve, whereas minimizing lambda-discrepancy lets the agent learn better policies.
- Theory: the lambda-discrepancy is exactly zero for all policies in MDPs (block MDPs give zero policy-spread difference) and is almost always non-zero for POMDPs, except for measure-zero policies.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the λ-discrepancy auxiliary loss to recurrent PPO | recurrent PPO · Battleship, PacMan, RockSample (11, 11), and RockSample (15, 15) | undiscounted return | Helped |
| add the λ-discrepancy auxiliary loss to recurrent PPO | recurrent PPO · large partially observable environments | undiscounted return | Helped |
| use widely separated λ values, with values close to 0 or 1 (ablation) | λ-discrepancy recurrent PPO · large partially observable environments | performance | Helped |
| remove the λ-discrepancy auxiliary loss while retaining two TD(λ) value functions (ablation) | two-value-function recurrent PPO · tested environments | performance | Hurt |
| learn a memory function by minimizing the λ-discrepancy | policy-gradient learning on classic POMDPs · classic POMDPs | normalized return | Helped |
| optimize a memoryless policy instead of learning a memory function (ablation) | Parity Check POMDP · Parity Check | normalized return | Hurt |
| add the λ-discrepancy auxiliary loss to an RNN | PacMan recurrent agent · PacMan | dot-location reconstruction | Helped |
| add a memory function to the value function | Parity Check POMDP · Parity Check | λ-discrepancy | Helped |
| use the proposed method rather than the baseline on classic POMDPs | baseline and λ-discrepancy approach · classic POMDPs | performance | No clear effect |
Surprising findings
- Optimal lambda values were close to either 0 or 1, i.e. large separation between lambda_1 and lambda_2 was beneficial.
- A PPO variant with two TD(lambda) value functions but without the lambda-discrepancy loss was never selected in the hyperparameter sweep.
Limitations
- Some POMDPs (e.g., Parity Check) have zero lambda-discrepancy for every policy, so non-Markovianity can go undetected.
- The deep RL auxiliary loss only approximates the lambda-discrepancy by comparing value estimates rather than converged fixed points.
- Theoretical guarantees hold only for value function fixed points, not for estimates before convergence.
Experimental setup
Datasets: Battleship PacMan RockSample (11,11) RockSample (15,15) T-maze Parity Check
Benchmarks: Battleship PacMan RockSample (11,11) RockSample (15,15)
Metrics: undiscounted return discounted return normalized return learning curves
Baselines: recurrent PPO (RNN) memoryless PPO PPO with two TD(lambda) value functions without auxiliary loss
Builds on: PPO generalized advantage estimation TD(lambda) recurrent PPO
Closest papers by meaning
- Partially Observable Reinforcement Learning with Memory Traces (ICML 2025 · similarity 0.91)
- Provable Representation with Efficient Planning for Partially Observable Reinforcement Learning (ICML 2024 · similarity 0.91)
- Provable Partially Observable Reinforcement Learning with Privileged Information (NeurIPS 2024 · similarity 0.90)
- When Your AIs Deceive You: Challenges of Partial Observability in Reinforcement Learning from Human Feedback (NeurIPS 2024 · similarity 0.88)
- Rethinking Transformers in Solving POMDPs (ICML 2024 · similarity 0.88)
- On Shallow Planning Under Partial Observability (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.