Research map / Reinforcement learning / Policy gradient and actor-critic
Parameterized Projected Bellman Operator
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the projected Bellman operator (PBO), a learned operator mapping value-function parameters to updated parameters, avoiding sample-based Bellman backups and projection steps in approximate value iteration, with offline (ProFQI) and online (ProDQN) algorithms.
Problem
Approximate value iteration relies on sample-based Bellman operator applications followed by costly projection onto a function space, where uninformative samples slow updates and projection adds expense.
Approach
Learn an operator acting directly on value-function parameters (PBO) that emulates the optimal Bellman operator followed by projection, so it can be iterated arbitrarily many times without new samples and its outputs always lie in the function space; parameterized with neural networks and trained with a loss over multiple PBO iterations (and optionally a fixed-point term).
Key results
- On chain-walk and LQR, ProFQI variants reduce approximation error as PBO applications increase, and beat FQI for the same number of Bellman iterations K.
- On car-on-hill after 9 iterations, ProFQI yields a policy close to optimal while FQI is significantly less accurate, with ProFQI performance remaining stable after K iterations.
- On bicycle balancing and lunar lander, ProDQN is stronger than DQN overall; on bicycle it improves slower initially but keeps increasing after K iterations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Learn and apply a parameterized projected Bellman operator instead of regular FQI updates | ProFQI · Car-on-hill | discounted cumulative reward | Helped |
| Learn and apply a parameterized projected Bellman operator instead of regular DQN updates | ProDQN · Bicycle balancing | discounted cumulative reward | Mixed |
| Learn and apply a parameterized projected Bellman operator instead of regular DQN updates | ProDQN · Lunar lander | discounted cumulative reward | Helped |
| Apply the trained PBO for additional iterations after training | ProFQI · Chain-walk | L2 norm to optimal action-value function | Helped |
| Add a fixed-point term to the PBO training loss (ablation) | ProFQI · Chain-walk, K=2 and K=5 | L2 norm to optimal action-value function | Helped |
| Add a fixed-point term to the PBO training loss (ablation) | ProFQI · Chain-walk, K=15 | L2 norm to optimal action-value function | Hurt |
| Use the closed-form chain-walk PBO with unknown transition and reward parameters (ablation) | ProFQI · Chain-walk, K=2 and K=5 | L2 norm to optimal action-value function | Helped |
| Use the closed-form LQR PBO with unknown system parameters (ablation) | ProFQI · LQR | distance to optimal action-value parameters | Helped |
| Use the closed-form LQR PBO with known system parameters (ablation) | ProFQI · LQR | distance to optimal action-value parameters | Helped |
Surprising findings
- On chain-walk with K=15, adding the fixed-point term slightly harmed approximation compared to ProFQI without it, attributed to linear PBO inadequacy.
Limitations
- Input and output size of PBO scales with the number of value-function parameters, making it hard to scale to large deep networks.
- Empirical evaluation limited to small-scale control tasks (chain-walk, LQR, car-on-hill, bicycle, lunar lander).
- Online exploration strategies based on PBO were not explored.
Experimental setup
Datasets: chain-walk LQR car-on-hill bicycle balancing lunar lander
Benchmarks: chain-walk LQR car-on-hill bicycle balancing lunar lander
Metrics: l2-norm distance to optimal action-value parameters discounted cumulative reward
Baselines: FQI DQN
Builds on: fitted Q-iteration deep Q-network approximate value iteration HyperNetworks
Closest papers by meaning
- Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learning (ICML 2025 · similarity 0.89)
- Natural Policy Gradient as Doubly Smoothed Policy Iteration: A Bellman-Operator Framework (NeurIPS 2026 · similarity 0.87)
- Computationally Efficient RL under Linear Bellman Completeness for Deterministic Dynamics (ICLR 2025 · similarity 0.87)
- Epistemic Bellman Operators (AAAI 2025 · similarity 0.86)
- Inverting the Bellman Equation: From $Q$-Values to World Models (NeurIPS 2026 · similarity 0.86)
- Deterministic Uncertainty Propagation for Improved Model-Based Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.