Research map / Reinforcement learning / Policy gradient and actor-critic

Intrinsic Benefits of Categorical Distributional Loss: Uncertainty-aware Regularized Exploration in Reinforcement Learning

NeurIPS 2025 · Reinforcement learning › Policy gradient and actor-critic · approach: Distributional value learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Decomposes the categorical distributional (KL) loss into a mean-related term and a distribution-matching entropy regularizer, and argues this uncertainty-aware regularization acts as an augmented reward that explains CDRL's advantage over classical RL.

Problem

Understanding the intrinsic theoretical benefits of categorical distributional RL (CDRL) over classical expectation-based RL, especially for practical algorithms like C51 and in actor-critic settings.

Approach

Using a histogram density estimator and a return density decomposition, the categorical KL loss splits into a mean-related term (equivalent to Neural FQI as bin width goes to zero) plus a cross-entropy regularizer against an induced density; this regularizer acts as an uncertainty-aware augmented reward in actor-critic, differing from MaxEnt's explicit action-entropy bonus.

Key results

• Decomposed C51 H(mu,q) degrades toward DQN as epsilon-related coefficient varies from 0.8 to 0.1 on most Atari games, supporting the regularizer as key to CDRL's advantage.
• AC+UE+VE (DSAC) outperforms both SAC and DAC on humanoidstandup but degrades on Ant and Swimmer, indicating interference between the two entropy regularizers.

Measured effects reported in the paper

Surprising findings

• Combining uncertainty-aware and vanilla entropy regularization can degrade performance on some MuJoCo tasks (Ant, Swimmer).

Limitations

• Analysis is mainly founded on categorical (C51-style) distributional RL; extension to general distributional RL such as quantile methods is left open.
• Neural FZI analysis does not give a formal convergence proof under an evolving data distribution.
• The convergence result for the mean-related term relies on the realizability assumption that the function class contains the targets.

Experimental setup

Datasets: Atari games MuJoCo tasks

Benchmarks: Atari games MuJoCo (humanoidstandup, Ant, Swimmer)

Metrics: learning curves / episodic return

Baselines: DQN C51 SAC DSAC IQN

Builds on: C51 Soft Actor-Critic Distributional SAC Neural FQI QR-DQN

Closest papers by meaning

• Categorical Distributional Reinforcement Learning with Kullback-Leibler Divergence: Convergence and Asymptotics (ICML 2025 · similarity 0.91)
• State Entropy Regularization for Robust Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
• Convergence Theorems for Entropy-Regularized and Distributional Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
• Near-Minimax-Optimal Distributional Reinforcement Learning with a Generative Model (NeurIPS 2024 · similarity 0.89)
• Distributional Reinforcement Learning with Regularized Wasserstein Loss (NeurIPS 2024 · similarity 0.88)
• Action Gaps and Advantages in Continuous-Time Distributional Reinforcement Learning (NeurIPS 2024 · similarity 0.88)