Research map / Reinforcement learning / Policy gradient and actor-critic

Enhancing Diversity In Parallel Agents: A Maximum State Entropy Exploration Story

ICML 2025 · Reinforcement learning › Policy gradient and actor-critic · approach: Regularized scalable deep value learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a parallel state-entropy maximization framework (PGPSE) where N parallel agents' mixture state distribution entropy is maximized via a centralized policy gradient, with concentration bounds and offline Q-learning gains on gridworlds.

Problem

Maximizing state entropy exploration when many identical agents collect data in parallel replicas of an environment, which yields redundant behavior; the question is how parallel agents should specialize their policies for efficient exploration in the finite-trial regime.

Approach

Treat parallel agents jointly by maximizing the entropy of the mixture state distribution induced by a uniform mixture of agent policies, balancing individual entropy with inter-agent KL diversity, and optimize it with a centralized REINFORCE-style policy gradient where each agent's score function is weighted by the team's empirical state entropy.

Key results

• Parallel agents (2, 4, 6) using PGPSE achieve higher normalized state entropy and larger support size than single agents with matched trajectories across Room and Maze gridworlds, over 5 seeds.
• Datasets collected by parallel PGPSE agents show higher empirical entropy than single-agent or random datasets.
• Offline Q-learning on parallel-agent datasets reaches goals across a larger number of goal states than datasets from a single max-entropy agent or a random policy.

Measured effects reported in the paper

Surprising findings

• Marginal benefit of adding more parallel agents diminishes as agent count grows, attributed to small state spaces.
• Parallel agents learn more deterministic individual policies than the single agent, which becomes more stochastic.

Limitations

• Evaluation limited to small discrete gridworld environments (43-state Room, 10x10 Maze).
• Single-trial (n=1) policy gradient with K=1 trajectory per agent; no continuous or high-dimensional experiments.
• Theoretical Frank-Wolfe convergence result relies on strong oracle assumptions.

Experimental setup

Datasets: gridworld Room and Maze environments (self-constructed)

Benchmarks: Room-det Room-stoc Maze-det Maze-stoc

Metrics: normalized state entropy support size offline Q-learning goal success rate

Baselines: single agent with K' trajectories random policy offline Q-learning on single-agent dataset

Builds on: state entropy maximization (Hazan et al.) finite-trials state entropy (Mutti et al.) REINFORCE policy gradient Parallel MDP

Closest papers by meaning

• The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networks (ICML 2025 · similarity 0.89)
• Towards Principled Unsupervised Multi-Agent Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
• Behavioral Entropy-Guided Dataset Generation for Offline Reinforcement Learning (ICLR 2025 · similarity 0.88)
• Novelty-Guided Data Reuse for Efficient and Diversified Multi-Agent Reinforcement Learning (AAAI 2025 · similarity 0.88)
• HyperMARL: Adaptive Hypernetworks for Multi-Agent RL (NeurIPS 2025 · similarity 0.87)
• Learning to Share: Selective Memory for Efficient Parallel Agentic Systems (ICML 2026 · similarity 0.87)