Research map / Reinforcement learning / Policy gradient and actor-critic

Quotient-Categorical Representations for Bellman-Compatible Average-Reward Distributional Reinforcement Learning

NeurIPS 2026 · Reinforcement learning › Policy gradient and actor-critic · approach: Distributional value learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a quotient-space categorical representation of the average-reward bias distribution (modulo common translation), proves non-expansiveness and fixed points of the projected operator, and gives TD-style stochastic approximation convergence results, with a five-state MDP experiment.

Problem

Distributional analogue of the average-reward bias is ill-posed on the real line because the bias is defined only up to an additive constant; need a Bellman-compatible distributional bias estimator with convergence guarantees.

Approach

Identify state-indexed law families up to a common translation (quotient space), use a shift-equivariant categorical projection on a fixed grid, and define a correctly centered projected operator that is non-expansive in a coordinate Cramér metric; then analyze asynchronous SKM TD recursions, with an online gain estimator for the unknown-gain case.

Key results

• Centered i.i.d. and coupled gain-estimated SKM recursions converge toward the exact KM fixed point on a five-state MRP, with the gain error |g_k - r̄| decreasing (Figure residual curves; no numeric values given in text).
• Theory: i.i.d. centered recursion achieves expected residual O(k^{-1/6}) up to log factor for step exponent near 2/3; Markovian rate O(k^{-1/10}) for exponent near 4/5.
• Wrong-centering ablation (g=0) yields a non-vanishing residual relative to the correctly centered operator G, showing fixed-grid coefficients depend on the representative.

Surprising findings

• Synchronous exact updates are gain-independent at the quotient-law level, yet fixed-grid categorical coefficients still depend on the gain choice after projection.
• Using raw rewards without centering does not yield a valid SA of the projected operator in general.

Limitations

• Analysis restricted to finite-state fixed-policy evaluation with fixed categorical supports.
• Non-expansive (not contractive) maps yield slower rates; Markovian rate is weaker (k^{-1/10}).
• Idealized centered-reward setting for main convergence results; control and function approximation not analyzed.

Experimental setup

Datasets: synthetic five-state MRP Pendulum-v1 (appendix)

Benchmarks: five-state MRP Pendulum-v1

Metrics: supremum-Cramér residual gain error

Baselines: exact KM iteration for projected operator centered i.i.d. SKM wrong-centering g=0 ablation

Builds on: categorical DRL (C51) stochastic Krasnoselskii-Mann iteration Cramér distance

Closest papers by meaning

• Categorical Distributional Reinforcement Learning with Kullback-Leibler Divergence: Convergence and Asymptotics (ICML 2025 · similarity 0.89)
• Foundations of Multivariate Distributional Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
• Distributional Bellman Operators over Mean Embeddings (ICML 2024 · similarity 0.88)
• Near-Minimax-Optimal Distributional Reinforcement Learning with a Generative Model (NeurIPS 2024 · similarity 0.88)
• Bellman Unbiasedness: Toward Provably Efficient Distributional Reinforcement Learning with General Value Function Approximation (ICML 2025 · similarity 0.88)
• Intrinsic Benefits of Categorical Distributional Loss: Uncertainty-aware Regularized Exploration in Reinforcement Learning (NeurIPS 2025 · similarity 0.88)