Research map / Reinforcement learning / MDPs and planning

Bellman Residual Minimization for Control: Geometry, Stationarity, and Convergence

NeurIPS 2026 · Reinforcement learning › MDPs and planning · approach: Average-reward and regularized MDPs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Analyzes Bellman residual minimization for control: shows the control Bellman residual (CBR) objective is locally Lipschitz, piecewise quadratic, with a Clarke subdifferential, and proves generalized gradient descent converges to stationary points; proposes a smooth soft variant (SCBR) with gradient descent and linear convergence in the tabular case.

Problem

Policy optimization via Bellman residual minimization is nonconvex and nonsmooth in the control setting due to the max operator, so its theory and optimization guarantees are largely unexplored, especially with linear function approximation.

Approach

Treats the control Bellman residual with the max operator as a piecewise quadratic, locally Lipschitz function, characterizes its Clarke subdifferential via active policies, and shows stationary points satisfy an oblique projected control Bellman equation; a soft (log-sum-exp) variant gives a C-infinity objective enabling gradient descent with linear rates in the tabular case.

Key results

• Tabular CBR stationary point is unique and equals Q*, and generalized gradient descent iterates converge to Q*.
• Tabular SCBR gradient descent converges exponentially fast to the soft Bellman fixed point under a PL condition.
• Error bound: ||Q_theta - Q*||_inf <= sqrt(2)/(1-gamma) * sqrt(f(theta)).

Surprising findings

• Stationary points of CBR with linear function approximation need not be unique, and the composite oblique operator need not be a contraction, yet a solution to the OP-CBE still exists.

Limitations

• Gradient-based methods are expected to be slower than value iteration in the tabular case.
• Stationary points under linear function approximation are not unique in general and error bounds are only partial.

Experimental setup

Metrics: Bellman residual sup-norm Q-function error

Baselines: projected value iteration value iteration

Builds on: Bellman residual gradient methods oblique projection framework for policy evaluation soft Q-learning

Closest papers by meaning

• Trajectory Bellman Residual Minimization: A Simple Value-Based Method for LLM Reasoning (NeurIPS 2025 · similarity 0.87)
• Computationally Efficient RL under Linear Bellman Completeness for Deterministic Dynamics (ICLR 2025 · similarity 0.87)
• Natural Policy Gradient as Doubly Smoothed Policy Iteration: A Bellman-Operator Framework (NeurIPS 2026 · similarity 0.86)
• Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learning (ICML 2025 · similarity 0.86)
• A Temporal Difference Method for Stochastic Continuous Dynamics (NeurIPS 2025 · similarity 0.85)
• Enhancing Value Function Estimation through First-Order State-Action Dynamics in Offline Reinforcement Learning (ICML 2024 · similarity 0.84)