Research map / Reinforcement learning / MDPs and planning
Is the Bellman residual a bad proxy?
Matthieu Geist, Bilal Piot, Olivier Pietquin
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Minimize the policy Bellman residual instead of maximizing mean value, using the distribution of interest | Garnet G(30,4,2) with F(8,3), ν = μ · 100 randomly generated Garnets | normalized value error | Hurt |
| Maximize mean value instead of minimizing Bellman residual, using the distribution of interest | Garnet G(30,4,2) with F(8,3) · 100 randomly generated Garnets | normalized value error | Improved |
| Use the ideal occupancy distribution d_{μ,π*} rather than μ for optimization | PS · Garnet, value error measured under μ | normalized value error | No clear effect |
| Use the ideal occupancy distribution d_{μ,π*} rather than μ for optimization | RPS · Garnet, value error measured under μ | normalized value error | No clear effect |
| Increase sampling-distribution mismatch from the ideal occupancy distribution | PS · Garnet with μ concentrated on one starting state; ν_α mixture | integrated normalized value error | No clear effect |
| Increase sampling-distribution mismatch from the ideal occupancy distribution | RPS · Garnet with μ concentrated on one starting state; ν_α mixture | integrated normalized value error | Hurt |
| Optimize the Bellman residual rather than mean value | RPS versus PS · Garnet with varying sampling distributions | integrated Bellman residual | Improved |
| Use the ideal occupancy distribution for RPS optimization while evaluating residual under μ | RPS · Garnet with ν = d_{μ,π*} | Bellman residual under μ | Hurt |
| Decrease the Bellman residual through policy residual search | RPS(μ) · Garnet with ν = μ | normalized value error | Improved |
| Increase mean value through policy search | PS(μ) · Garnet with ν = μ | Bellman residual | Mixed |
Closest papers by meaning
- Bellman Residual Minimization for Control: Geometry, Stationarity, and Convergence (NeurIPS 2026 · similarity 0.91)
- Trajectory Bellman Residual Minimization: A Simple Value-Based Method for LLM Reasoning (NeurIPS 2025 · similarity 0.88)
- Natural Policy Gradient as Doubly Smoothed Policy Iteration: A Bellman-Operator Framework (NeurIPS 2026 · similarity 0.87)
- Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learning (ICML 2025 · similarity 0.87)
- Computationally Efficient RL under Linear Bellman Completeness for Deterministic Dynamics (ICLR 2025 · similarity 0.86)
- Finite-Sample Analysis of Policy Evaluation for Robust Average Reward Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
