Research map / Reinforcement learning / Policy gradient and actor-critic
High Entropy Regularization Leads to Symmetry Equivariant Policies in Dec-POMDPs
Johannes Forkel, Constantin Ruhdorfer, Michael Beukman, Andreas Bulling, Jakob Foerster
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's abstract only. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proves that in any Dec-POMDP, sufficiently high entropy regularization makes tabular softmax policy gradient flow converge to a unique joint policy that is equivariant to all Dec-POMDP symmetries, so independently trained policies are cross-play compatible.
Problem
Independently trained policies in decentralized cooperative settings (Dec-POMDPs) often fail to be compatible in cross-play because learning converges to arbitrary, symmetry-breaking conventions.
Approach
Shows that sufficiently high entropy regularization makes policy gradient flow with tabular softmax parametrization converge from any initialization to the same symmetry-equivariant joint policy, yielding cross-play compatibility as a theoretical guarantee.
Key results
- Proves that for sufficiently high entropy regularization, policy gradient flow with tabular softmax parametrization converges from any initialization to the same joint policy in any Dec-POMDP.
- The limiting joint policy is equivariant with respect to all symmetries of the Dec-POMDP.
- Policies from different initializations are fully compatible in cross-play, per the abstract's claim.
Limitations
- Results hold only for sufficiently high entropy regularization; the abstract does not characterize the threshold's practical size.
- Analysis is for tabular softmax parametrization, not neural function approximation.
Closest papers by meaning
- ${\rm E}(3)$-Equivariant Actor-Critic Methods for Cooperative Multi-Agent Reinforcement Learning (ICML 2024 · similarity 0.87)
- Flow Matching Policy Optimization with Mirror Descent and Entropy Constraints (NeurIPS 2026 · similarity 0.86)
- Generative Actor-Critic with Soft Bridge Policies (NeurIPS 2026 · similarity 0.86)
- Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matching (NeurIPS 2026 · similarity 0.86)
- Convergence Theorems for Entropy-Regularized and Distributional Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
- How to Explore with Belief: State Entropy Maximization in POMDPs (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
