Research map / Reinforcement learning / Policy gradient and actor-critic

Quantile Geometry Regularization for Distributional Reinforcement Learning

NeurIPS 2026 · Reinforcement learning › Policy gradient and actor-critic · approach: Distributional value learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RQIQN, an IQN variant that adds a closed-form Wasserstein distributionally robust quantile correction to the TD residual, evaluated on risk-sensitive surface-vehicle navigation and Atari.

Problem

Quantile-based distributional RL (e.g., IQN) learns return distributions via bootstrapped quantile regression, but finite quantile statistics are not Bellman-closed, so target quantiles can distort or collapse the learned return distribution.

Approach

Interprets each IQN loss slot as a local empirical quantile estimation problem and robustifies it against a type-p Wasserstein ambiguity set, yielding a closed-form, fraction-dependent, median-antisymmetric and monotone location correction added to the TD residual, with the radius decayed over training by a reverse-logistic schedule.

Key results

• In hard-mode navigation, RQIQN reaches 83.24% success vs 64.80% for IQN, and 85.64% with adaptive CVaR sampling (5 seeds, 500 episodes each).
• RQIQN improves over IQN on most Atari games, notably Alien, BattleZone, Frostbite, and Qbert, but not NameThisGame (3 seeds).
• Ablation on Qbert: check-loss with p=2 is best; p=infinity variants degrade performance, showing sensitivity to Wasserstein geometry.

Surprising findings

• On NameThisGame, advanced DistRL methods fluctuate and converge to suboptimal returns, while C51 remains stable.
• The p=infinity Wasserstein ambiguity set degrades performance for both check-loss and Huber variants.

Limitations

• Closed-form correction for p=2 is an ad hoc bounded adaptation, not the exact WDRQR formula, and the check-loss theory does not directly carry to the Huber loss used in practice.
• Atari results are preliminary, with only 3 seeds and a subset of 9 games; some games (NameThisGame) show degradation.
• Requires tuning of epsilon_0, k, and t0 schedule per environment.

Experimental setup

Datasets: Atari-5 subbenchmark Atari games (ALE, sticky actions) custom marine navigation simulator with vortex flows

Benchmarks: Atari-5 / Atari games (9 games) ASV navigation hard-mode (10 obstacles, 8 vortex cores)

Metrics: score / return success rate collision rate timeout rate time to success energy to success

Baselines: DQN IQN C51 QR-DQN Rainbow IEQN ER-DQN

Builds on: IQN QR-DQN WDRQR (Wasserstein distributionally robust quantile regression) Dopamine

Closest papers by meaning

• Implicit Quantile Networks for Distributional Reinforcement Learning (ICML 2018 · similarity 0.92)
• Fully Parameterized Quantile Function for Distributional Reinforcement Learning (NeurIPS 2019 · similarity 0.91)
• Tilted Quantile Gradient Updates for Quantile-Constrained Reinforcement Learning (AAAI 2025 · similarity 0.90)
• Quantile-Coupled Flow Matching for Distributional Reinforcement Learning (NeurIPS 2026 · similarity 0.86)
• Intrinsic Benefits of Categorical Distributional Loss: Uncertainty-aware Regularized Exploration in Reinforcement Learning (NeurIPS 2025 · similarity 0.85)
• Quotient-Categorical Representations for Bellman-Compatible Average-Reward Distributional Reinforcement Learning (NeurIPS 2026 · similarity 0.85)