Research map / Reinforcement learning / Policy gradient and actor-critic
Dual Critic Reinforcement Learning under Partial Observability
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DCRL, a dual-critic actor-critic method for POMDPs that mixes an oracle critic (history+state) with a standard history-only critic via a clipped weighting, claiming lower variance while staying unbiased.
Problem
Reinforcement learning under partial observability, where asymmetric critics that use full state during training can reduce sample complexity but suffer high variance in practice.
Approach
Combines an oracle critic V(h,s) and a standard critic V(h) through a dual value/advantage weighted by beta, with beta clipped to zero when the advantage is non-positive; theoretically shows the dual estimator stays unbiased and has variance at most that of the oracle critic.
Key results
- On MiniGrid after 1e5 frames with A2C, DCRL scores 0.63±0.07 vs 0.39 best baseline at 1e6 frames for Unbiased Asymmetric Actor-Critic (DCRL 0.89±0.01 at 1e6 frames).
- DCRL outperforms all baselines in 26 of 27 MiniGrid games per the paper.
- DCRL enhances PPO performance across all four MiniWorld environments (TMaze, YMaze, WallGap, Sidewalk).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DCRL instead of the strongest compared baseline | A2C · MiniGrid, 1e5 frames (27 games) | average score: 0.31±0.08 -> 0.63±0.07 | Helped |
| Use DCRL instead of the strongest compared baseline | A2C · MiniGrid, 1e6 frames (13 games) | average score: 0.39±0.09 -> 0.89±0.01 | Helped |
| Use DCRL instead of the strongest compared baseline | PPO · MiniGrid, 1e5 frames (27 games) | average score: 0.28±0.10 -> 0.47±0.08 | Helped |
| Use DCRL instead of the strongest compared baseline | PPO · MiniGrid, 1e6 frames (13 games) | average score: 0.60±0.06 -> 0.84±0.03 | Helped |
| Replace the dual-critic structure with a standard critic only (ablation) | DCRL on MiniGrid · DistShift2, LavaCrossingS9N2, LavaGapS6, MultiRoom-N2-S4 | return | Hurt |
| Replace the dual-critic structure with an oracle critic only (ablation) | DCRL on MiniGrid · DistShift2, LavaCrossingS9N2, LavaGapS6, MultiRoom-N2-S4 | return | Hurt |
| Interchange the critics' roles in the weighting mechanism (ablation) | DCRL on MiniGrid · DistShift2, LavaCrossingS9N2, LavaGapS6, MultiRoom-N2-S4 | return | Mixed |
| Use state as the standard critic's input instead of history (ablation) | DCRL on MiniGrid · DistShift2, LavaCrossingS9N2, LavaGapS6, MultiRoom-N2-S4 | return | Mixed |
| Remove clipping from the dynamic weighting mechanism and use fixed critic weights (ablation) | DCRL on MiniGrid · MiniGrid environments in the β ablation | return | Mixed |
| Vary β across {1/5, 1/3, 1/2, 2/3, 4/5} (ablation) | DCRL on MiniGrid · MiniGrid environments | return | No clear effect |
Surprising findings
- The Unbiased Asymmetric Actor-Critic performs worse than the Recurrent Actor-Critic across all MiniWorld tasks, where state images are noisy.
- Providing complete state information can sometimes hinder performance in certain environments, per the authors' limitations discussion.
Limitations
- Assumes complete state information is available during training; partial state access is not studied.
- Performance can degrade when state information is noisy or one-to-one with observations, making the oracle critic variance-free and DCRL ineffective.
Experimental setup
Datasets: MiniGrid MiniWorld
Benchmarks: MiniGrid MiniWorld
Metrics: average return mean and standard error over seeds
Baselines: Recurrent Actor-Critic Asymmetric Actor-Critic Oracle Guiding Unbiased Asymmetric Actor-Critic DreamerV3 DCRL Interchange Version DCRL State Version
Builds on: Unbiased Asymmetric Actor-Critic Advantage Actor-Critic Proximal Policy Optimization Self-Imitation Learning
Closest papers by meaning
- A Theoretical Justification for Asymmetric Actor-Critic Algorithms (ICML 2025 · similarity 0.88)
- Provable Partially Observable Reinforcement Learning with Privileged Information (NeurIPS 2024 · similarity 0.88)
- Mitigating Partial Observability in Sequential Decision Processes via the Lambda Discrepancy (NeurIPS 2024 · similarity 0.88)
- Partially Observable Reinforcement Learning with Memory Traces (ICML 2025 · similarity 0.87)
- Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL (ICML 2024 · similarity 0.87)
- Provable Representation with Efficient Planning for Partially Observable Reinforcement Learning (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.