Research map / Reinforcement learning / Policy gradient and actor-critic
A Principled Path to Fitted Distributional Evaluation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a principled framework for fitted distributional evaluation (FDE), extending fitted Q-evaluation to return distributions via functional Bregman divergences, with convergence analysis in tabular and non-tabular settings.
Problem
Distributional off-policy evaluation requires extending fitted Q-evaluation to return distributions, but there is no general guidance on which discrepancy function to use, and existing analyses are limited (mostly tabular, bounded returns).
Approach
Three principles (contraction-inducing metric, unbiased population minimizer via functional Bregman divergence, and metric-control of the objective) identify valid discrepancies; functional Bregman divergences (tied to strictly proper scoring rules) make the conditional expectation of the distributional Bellman backup the minimizer, yielding FDE methods with convergence rates in general state-acti
Key results
- On Atari with N=10K and M=200, FDE methods (KL, Energy) reduce W1 inaccuracy over iterations and outperform FLE, QRDQN and IQN in most games.
- TVD-based FDE shows no accuracy improvement across iterations, supporting the need for functional Bregman divergences.
- Cramer FDE and Energy FDE give convergence rates for W_p-type errors; the Cramer rate N^{-1/(p(2+alpha))} is faster than FLE's N^{-(1-alpha)/(2p)} in the bounded 1-D case.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FDE with KL, Energy, PDF-L2, or RBF instead of FLE | Distributional OPE on LQR · LQR | performance | Helped |
| Use FDE methods, particularly KL or Energy, instead of baseline methods | Distributional OPE on Atari games · Atari games | mean inaccuracy | Helped |
| Use FDE methods instead of baseline methods | Distributional OPE on Atari games · Atari games | inaccuracy | Helped |
| Use TVD as the FDE objective | Distributional OPE on Atari games · Atari games | inaccuracy across iterations | No clear effect |
| Use Hyvärinen divergence as the FDE objective | FDE · Atari games | performance | Helped |
Surprising findings
- Total variation distance based FDE shows no improvement in accuracy through iterations.
- Some natural metrics that guarantee contraction (e.g., Wasserstein distance with sample-based objective) fail due to biased gradients.
Limitations
- Completeness assumption may be overly restrictive in practice.
- Theory assumes data splitting so samples are independent across iterations; data reuse not analyzed.
- Necessity of functional Bregman divergence for the unbiased-minimizer property is open.
Experimental setup
Datasets: Atari games (Atlantis, Breakout, Enduro, KungFuMaster, Pong, Qbert, SpaceInvaders) offline data linear quadratic regulator simulations
Benchmarks: Atari games (7 games) linear quadratic regulator
Metrics: W1 inaccuracy
Baselines: FLE QRDQN IQN TVD-based FDE
Builds on: fitted Q-evaluation distributional Bellman operator MMDRL fitted likelihood evaluation QR-DQN IQN
Closest papers by meaning
- Distributional Reinforcement Learning With Quantile Regression (AAAI 2018 · similarity 0.88)
- Variance-Aware Off-Policy Evaluation with Linear Function Approximation (NeurIPS 2021 · similarity 0.88)
- Bootstrapping Fitted Q-Evaluation for Off-Policy Inference (ICML 2021 · similarity 0.88)
- Intrinsically Efficient, Stable, and Bounded Off-Policy Evaluation for Reinforcement Learning (NeurIPS 2019 · similarity 0.88)
- Dynamic Uncertainty Estimation for Offline Reinforcement Learning (AAAI 2025 · similarity 0.87)
- Conformal Prediction Beyond the Horizon: Distribution-Free Inference for Policy Evaluation (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.